One of the key use cases for generative AI involves answering questions over private datasets, with retrieval-augmented generation (RAG) as the go-to framework. As new RAG techniques emerge, there’s a growing need to benchmark their performance across diverse datasets and metrics.  To meet this need, we’re introducing BenchmarkQED, a new suite of tools that automates RAG benchmarking at scale, available on GitHub (opens in new tab). It includes components for query generation, evaluation, and dataset preparation, each designed to support rigorous, reproducible testing.   BenchmarkQED complements the RAG methods in our open-source GraphRAG library, enabling users to run a GraphRAG-style evaluation across models, metrics, and datasets. GraphRAG uses a large language model (LLM) to generate and summarize entity-based knowledge graphs, producing more comprehensive and diverse answers than standard RAG for large-scale tasks.  In this post, we walk through the core components of BenchmarkQED that contribute to the overall benchmarking process. We also share some of the latest benchmark results comparing our LazyGraphRAG system to competing methods, including a vector-based RAG with a 1M-token context window, where the leading LazyGraphRAG configuration showed significant win rates across all combinations of quality metrics and query classes. In the paper, we distinguish between local queries, where answers are found in a small number of text regions, and sometimes even a single region, and global queries, which require reasoning over large portions of or even the entire dataset.  Conventional vector-based RAG excels at local queries because the regions containing the answer to the query resemble the query itself and can be retrieved as the nearest neighbor in the vector space of text embeddings. However, it struggles with global questions, such as, “What are the main themes of the dataset?” which require understanding dataset qualities not explicitly stated in the text.   AutoQ: Automated query synthesis This limitation motivated the development of GraphRAG a system designed to answer global queries. GraphRAG’s evaluation requirements subsequently led to the creation of AutoQ, a method for synthesizing these global queries for any dataset. AutoQ extends this approach by generating synthetic queries across the spectrum of queries, from local to global. It defines four distinct classes based on the source and scope of the query (Figure 1, top) forming a logical progression along the spectrum (Figure 1, bottom). Figure 1. Construction of a 2×2 design space for synthetic query generation with AutoQ, showing how the four resulting query classes map onto the local-global query spectrum.  AutoQ can be configured to generate any number and distribution of synthetic queries along these classes, enabling consistent benchmarking across datasets without requiring user customization. Figure 2 shows the synthesis process and sample queries from each class, using an AP News dataset. Figure 2. Synthesis process and example query for each of the four AutoQ query classes.  About Microsoft Research Advancing science and technology to benefit humanity View our story Opens in a new tab AutoE: Automated evaluation framework  Our evaluation of GraphRAG focused on analyzing key qualities of answers to global questions. The following qualities were used for the current evaluation: Comprehensiveness: Does the answer address all relevant aspects of the question?  Diversity: Does it present varied perspectives or insights?  Empowerment: Does it help the reader understand and make informed judgments?  Relevance: Does it address what the question is specifically asking?   The AutoE component scales evaluation of these qualities using the LLM-as-a-Judge method. It presents pairs of answers to an LLM, along with the query and target metric, in counterbalanced order. The model determines whether the first answer wins, loses, or ties with the second. Over a set of queries, whether from AutoQ or elsewhere, this produces win rates between competing methods. When ground truth is available, AutoE can also score answers on correctness, completeness, and related metrics. An illustrative evaluation is shown in Figure 3. Using a dataset of 1,397 AP News articles on health and healthcare, AutoQ generated 50 queries per class (200 total). AutoE then compared LazyGraphRAG to a competing RAG method, running six trials per query across four metrics, using GPT-4.1 as a judge. These trial-level results were aggregated using metric-based win rates, where each trial is scored 1 for a win, 0.5 for a tie, and 0 for a loss, and then averaged to calculate the overall win rate for each RAG method. Figure 3. Win rates of four LazyGraphRAG (LGR) configurations across methods, broken down by the AutoQ query class and averaged across AutoE’s four metrics: comprehensiveness, diversity, empowerment, and relevance. LazyGraphRAG outperforms comparison conditions where the bar is above 50%. The four LazyGraphRAG conditions (LGR_b200_c200, LGR_b50_c200, LGR_b50_c600, LGR_b200_c200_mini) differ by query budget (b50, b200) and chunk size (c200, c600). All used GPT-4o mini for relevance tests and GPT-4o for query expansion (to five subqueries) and answer generation, except for LGR_b200_c200_mini, which used GPT-4o mini throughout. Comparison systems were GraphRAG (Local, Global, and Drift Search), Vector RAG with 8k- and 120k-token windows, and three published methods: LightRAG (opens in new tab), RAPTOR (opens in new tab), and TREX (opens in new tab). All methods were limited to the same 8k tokens for answer generation. GraphRAG Global Search used level 2 of the community hierarchy. LazyGraphRAG outperformed every comparison condition using the same generative model (GPT-4o), winning all 96 comparisons, with all but one reaching statistical significance. The best overall performance came from the larger budget, smaller chunk size configuration (LGR_b200_c200). For DataLocal queries, the smaller budget (LGR_b50_c200) performed slightly better, likely because fewer chunks were relevant. For ActivityLocal queries, the larger chunk size (LGR_b50_c600) had a slight edge, likely because longer chunks provide a more coherent context. Competing methods performed relatively better on the query classes for which they were designed: GraphRAG Global for global queries, Vector RAG for local queries, and GraphRAG Drift Search, which combines both strategies, posed the strongest challenge overall. Increasing Vector RAG’s context window from 8k to 120k tokens did not improve its performance compared to LazyGraphRAG. This raised the question of how LazyGraphRAG would perform against Vector RAG with 1-million token context window containing most of the dataset. Figure 4 shows the follow-up experiment comparing LazyGraphRAG to Vector RAG using GPT-4.1 that enabled this comparison. Even against the 1M-token window, LazyGraphRAG achieved higher win rates across all comparisons, failing to reach significance only for the relevance of answers to DataLocal queries. These queries tend to benefit most from Vector RAG’s ranking of directly relevant chunks, making it hard for LazyGraphRAG to generate answers that have greater relevance to the query, even though these answers may be dramatically more comprehensive, diverse, and empowering overall. Figure 4. Win rates of LazyGraphRAG (LGR) over Vector RAG across different context window sizes, broken down by the four AutoQ query classes and four AutoE metrics: comprehensiveness, diversity, empowerment, and relevance. Bars above 50% indicate that LazyGraphRAG outperformed the comparison condition.  AutoD: Automated data sampling and summarization Text datasets have an underlying topical structure, but the depth, breadth, and connectivity of that structure can vary widely. This variability makes it difficult to evaluate RAG systems consistently, as results may reflect the idiosyncrasies of the dataset rather than the system’s general capabilities. The AutoD component addresses this by sampling datasets to meet a target specification, defined by the number of topic clusters (breadth) and the number of samples per cluster (depth). This creates consistency across datasets, enabling more meaningful comparisons, as structurally aligned datasets lead to comparable AutoQ queries, which in turn support consistent AutoE evaluations. AutoD also includes tools for summarizing input or output datasets in a way that reflects their topical coverage. These summaries play an important role in the AutoQ query synthesis process, but they can also be used more broadly, such as in prompts where context space is limited. Since the release of the GraphRAG paper, we’ve received many requests to share the dataset of the Behind the Tech (opens in new tab) podcast transcripts we used in our evaluation. An updated version of this dataset is now available in the BenchmarkQED repository (opens in new tab), alongside the AP News dataset containing 1,397 health-related articles, licensed for open release.   We hope these datasets, together with the BenchmarkQED tools (opens in new tab), help accelerate benchmark-driven development of RAG systems and AI question-answering. We invite the community to try them on GitHub (opens in new tab).  Opens in a new tab

Next week, the former president of “Grant Theft Auto” maker Rockstar North launches his first title since leaving the Take-Two Interactive-owned video game developer and opening his own studio, Build A Rocket Boy: the AAA narrative-driven action-adventure thriller “MindsEye.” Published by IOI Partners, the team behind the “Hitman” franchise, the Unreal Engine 5-built game will debut June 10 across PlayStation 5, Xbox Series X and S, and on PC via Steam and Epic Games Store with a $59.99 price tag for the standard edition. Related Stories Set in the near-futuristic city of Redrock, “MindsEye” puts players into the role of Jacob Diaz, a former soldier haunted by fragmented memories from his mysterious MindsEye neural implant, as he uncovers a conspiracy involving rogue AI, corporate greed, an unchecked military, and a threat so sinister that it endangers the very survival of humanity. Popular on Variety But the base story isn’t the biggest draw for “MindsEye,” which includes Build A Rocket Boy’s proprietary Game Creation System, that enables players to, well, “craft anything in their minds eye.” Per the studio, “Players can craft their own experiences using all of the ‘MindsEye’ assets, creating everything from custom missions to entirely new scenarios within the game’s expansive, richly detailed world. Whether you’re designing a high-speed chase through Redrock’s bustling cityscapes or a stealth mission in its industrial outskirts, it is designed to be intuitive and easy to use, ensuring that players of all skill levels can bring their imagination to life.” Benzies’ Edinburgh-based Build A Rocket Boy has promised “fresh premium content” will rollout monthly for the game, including regular releases of new missions, challenges and game assets. While “MindsEye” is the first title from Benzies since he launched BARB after leaving Rockstar in 2016 (Benzies was the lead “Grand Theft Auto” developer across the third through fifth games in the franchise, as well as “Grand Theft Auto Online,” and was in a legal battle with parent company Take Two over unpaid royalties from 2016 until 2019), it’s just step one in the prolific producer’s plan to shake up the gaming industry. “At Build A Rocket Boy, our vision goes far beyond a single title,” Benzies told Variety. “‘MindsEye’ is the first episode and central story around which ever-expanding interconnected episodes will span. We’re already working on future episodes, which will introduce alternate realities while maintaining it’s core themes of hope, redemption, and the intrigue of civilizations past and future, drawing from the lore and multiverse concepts.” See Variety‘s full interview with Benzies below, including the inevitable comparisons that will be drawn between “MindsEye” and the aesthetic of the “GTA” franchise, and his hopes for Rockstar Games’ highly anticipated and much-delayed “GTA 6.” Where did the concept for “MindsEye” come from? I pull a lot of inspiration from the real world. Watching the actions of humans – their foibles and their virtues. Watching the advancement of technology and how we adapt, or indeed, do not adapt. We’ve been moving to an automated world for many years now, and the impact on humans, especially with recent advancements in AI, which serves as good fodder for a story and even better for a video game. I think we all have this little nagging feeling about how humans and AI will blend together in the future—will it go smoothly, or will it turn sinister? We’re fans of all different types of media, and we’ve drawn influence from cinematic visionaries like Ridley Scott, Paul Greengrass, Christopher Nolan, and J.J. Abrams, and films like “The Bourne Identity,” “Memento,” and TV series “Lost” — they’re all exploring memory, perception, and control in their own ways. So, while we nod to those influences here and there, we wanted to build something that feels fresh, grounded in today’s world, but still asking the kinds of questions that have always made this genre powerful. With your “GTA” roots, obvious comparisons are already being drawn between the style and aesthetic of that franchise and “MindsEye.” Comparisons will always be made—it’s the way human beings pigeonhole concepts. But “MindsEye” isn’t built to fit into anyone else’s box. Many games share the same core elements: cars, guns, cities, and charismatic characters, and differentiation is even tougher in today’s entertainment landscape. Streaming, social media, and on-demand binge culture have fractured attention spans, and consumer mindshare is a brutal battlefield for all IP. Our industry continues to celebrate each other’s breakthroughs, and I’m proud that our collective innovation is advancing the medium of gaming, even if our paths diverge. As an independent studio we have the freedom to break ground in experimental new ways and the challenge is balancing innovation with familiarity—too much “new” risks alienating fans, too much “same” feels stale. It’s about nailing what makes your game’s world feel alive and urgent. “MindsEye” is about consequence and connection—it’s cinematic, reactive, and meant to feel like a world you’re not just playing in, but able to create in it too. We’re excited to see what they’ve crafted with “GTA VI ,” and I can’t wait to play it as a consumer for the first time. They’re always delivering something new, unique and at a scale that very few can pull off. What does MindsEye represent in BARB’s larger vision and long-term strategy? Are you plotting this out as a multi-game franchise or your first standalone? At Build A Rocket Boy, our vision goes far beyond a single title. “MindsEye” is the first episode and central story around which ever-expanding interconnected episodes will span. We’re already working on future episodes, which will introduce alternate realities while maintaining it’s core themes of hope, redemption, and the intrigue of civilizations past and future, drawing from the lore and multiverse concepts. It’s the future of entertainment to allow active participation so players feel like they have agency and can immerse themselves in our world as they want to. We are introducing three products in one game that will revolutionize AAA-quality interactive gaming and storytelling: “MindsEye” narrative story, Play.MindsEye, and Build.MindsEye. In our tightly crafted action-noir, “MindsEye” narrative story we have rips in time accessed through portals at strategic points throughout the game – so while you play as Jacob Diaz on his personal journey, players can also explore side stories and delve deeper into the backstories of characters they encounter along the way. In this way we are delivering companion content at the same time as the anchor content, weaving a rich narrative tapestry which will continue to evolve and expand giving greater depth to characters so you understand their personality and motivations. How do digital products Play.MindsEye (formerly named Arcadia) and Build.MindsEye (formerly Everywhere) tie in to plans for “MindsEye” and what BARB wants to offer gamers? In this new era of entertainment, where streaming platforms, boom-and-bust games, and an on-demand culture dominate, we’re pushing things in a new direction—with an interface that simplifies how we consume not just games, but all forms of entertainment. Consumers are moving away from 2D browsing into fully 3D, immersive experiences. Put simply, we’re shifting from passive interaction to active participation. As with all new products, things evolve. Arcadia was originally envisioned as our creation platform, but as we continued developing “MindsEye” and building out BARB’s ecosystem, it naturally grew into something more focused— Play.MindsEye and Build.MindsEye. Play delivers cinematic, high-intensity gameplay with missions and maps that constantly evolve. Build gives players intuitive tools to create their own content—no technical skills required, just imagination and intent. For BARB to fully realize our vision, we had to beta test our creation system with a community of builders in real-time and started with Everywhere while we were in stealth mode developing MindsEye. How did you settle on IOI as publishing partner? We’ve always found the way IOI handled the “Hitman” franchise interesting. They are one of the few publishers that have taken their single-player IP and increased their player count and amplified their community culture over time. From a technology point of view, their one executable approach for all of their content is very smart, and we always planned to have a similar approach, which encouraged us to join forces. This interview has been edited and condensed.

Nintendo has predicted it will sell 15 million Switch 2s in its current financial year. Analysts think the number is conservative; Nintendo says the price of the Switch 2 is what’s holding back that estimate. But what does that mean, really? Is it a reasonable target? If Nintendo hits it, does that guarantee the Switch 2 will be a massive hit? If a console sells out at launch, what does that tell us? Is there still a chance of a Wii U-style flop?It’s impossible to know for sure, but a look at all of Nintendo’s past console launches can provide some clues. I’ve dug deep into past Nintendo sales figures to determine which Nintendo consoles enjoyed the best launches. To get a more reliable picture than that initial, almost inevitable sellout, I’ve defined the launch as the console’s first year (or rather, first four financial quarters) on the market.Note that Nintendo only started reporting quarterly sales for its systems in the mid-2000s, and the earliest data is annual at best and hard to come by, so some of these numbers are approximate. Note, too, that older systems had staggered launches across the three major markets (Japan, North America, and Europe), sometimes over several years, slowing down their potential sales.Still, there are some surprising results here that put Nintendo’s 15 million forecast for Switch 2 in context. Selling that many units would definitely not be bad news — but it doesn’t indicate a slam-dunk, either.Image: James Bareham/Polygon1. Game Boy AdvanceFirst four quarters: approx. 18.1 millionLifetime: 81.51 million (5th overall)Release: March-June 20012. 3DSFirst four quarters: 15.03 millionLifetime: 75.94 million (6th)Release: February-March 2011There’s a clear pattern to Nintendo’s two fastest sellers; they were successors to massive hits (the Game Boy and DS) in the handheld market, where Nintendo enjoyed total dominance. Nintendo was so bullish about Game Boy Advance, following the decade-long reign of the Game Boy format, that it forecast an astonishing 24 million sales in its first year, while 3DS followed Nintendo’s biggest seller ever. Both sold well, but neither quite lived up to their forebears.Photo: Nintendo3. SwitchFirst four quarters: 14.86 millionLifetime: 152.12 million (2nd, for now)Release: March 20174. WiiFirst four quarters: 13.17 millionLifetime: 101.63 million (4th)Release: November-December 2006The Switch and the Wii are the only Nintendo consoles with sales over 10 million in the first year and over 100 million in their lifetimes. Quarter for quarter, they both sold incredibly consistently over time. This is surely what Nintendo would like all its hardware launches to look like, and what it’s hoping for with the Switch 2.5. DSFirst four quarters: 8.83 millionLifetime: 154.02m (1st, for now)Release: November 2004-March 2005It’s a surprise DS didn’t launch more strongly, considering the runaway early sales of its handheld predecessor, Game Boy Advance. But its launch games weren’t the best and it took a while for the console’s strange design to find its eventual massive casual audience. In its fifth quarter — holiday 2005 — it suddenly took off.Shigeru Miyamoto proudly regards his purplest child, the GameCube. Photo: Lucy Nicholson/AFP via Getty Images6. GameCubeFirst four quarters: approx. 6.7 millionLifetime: 21.74 million (10th)Release: Sep 2001-May 20027. Nintendo 64First four quarters: 5.80 millionLifetime: 32.93 million (9th)Release: June 1996-March 1997The two Nintendo home consoles made during PlayStation’s ascendency enjoyed reasonably strong launches but petered out due to a lack of software support. GameCube sold 30% of its lifetime total in its first year on sale — an unfortunate achievement that beats even Wii U’s dismal ratio.Photo: James Bareham/Polygon8. Game BoyFirst four quarters: 3.93 millionLifetime: 118.69 million (3rd)Release: April 1989-September 1990The legend of Game Boy is that it was an instant smash, thanks to the high-stakes acquisition of Tetris. But while its launch was healthy for the time, it didn’t begin to indicate what Nintendo’s defining handheld would achieve — sales in the first year (before it reached Europe) were just 3% of what it would go on to sell across Game Boy and Game Boy Color.9. Wii UFirst four quarters: 3.91 millionLifetime: 13.56 million (11th)Release: November-December 2012Nintendo’s most recent (but not its worst) flop actually started quite strongly, shifting over 3 million units in its first quarter — but then suffered a catastrophic drop-off, selling less than 900,000 worldwide during the rest of its first year on sale. Looking past the initial holiday sell-out, the signs of the disaster to come were clear.Nintendo paid tribute to its two trailblazing home systems with the NES and SNES Classic mini-consoles. Image: Shai Ben-Dor/Vox MediaFirst four quarters: approx. 2.9 million (mostly Japan)Lifetime: 49.1 million (8th)Release: November 1990-June 1992 Launch sales: 2.5 million by end of 1984 (Japan only)Lifetime: 61.91 million (7th)Release: July 1983 to 1987 and laterBoth Nintendo’s early home consoles launched strongly in Japan but took a long time to take off in the West. The NES didn’t fully launch in the U.S. and Europe until 1986, amid caution after the 1980s video game crash, while the SNES was beaten to market in the West by the Sega Genesis and suffered as a result. 12. Virtual BoyLaunch and lifetime sales: 770,000Release: July-August 1995You can’t get a worse launch than being discontinued within a year of going on sale!—What does this mean for Switch 2’s launch?If Nintendo meets or exceeds its target of 15 million Switch 2s sold in its first financial year, it will rank among the top Nintendo console launches ever. A Wii U-style flop looks very unlikely, unless sales drop off sharply after its first few months.But the most telling comparisons here are Nintendo’s top two launches: Game Boy Advance and 3DS. Like Switch 2, they were both conservative, easy-to-understand sequels to huge sellers in a market sector Nintendo had total control of. And while both reached respectable lifetime totals, they got nowhere near the lifetime sales of their more innovative predecessors.Could this be Switch 2’s fate? Judging by Nintendo’s launch history, it’s possible — maybe even likely.See More:

Key Takeaways AMD’s RX 9060 XT is set to launch on June 5th, 2025 in both 8GB and 16GB versions under the same name, creating confusion and backlash. Reviewers and gamers say 8GB of VRAM isn’t enough for modern gaming, especially at 1440p. AMD’s decision to showcase only the 16GB model in benchmarks raised concerns about transparency. This move mirrors Nvidia’s controversial RTX 4060 Ti rollout, suggesting an industry trend of misleading GPU marketing. It all started with a new GPU announcement. The AMD Radeon RX 9060 XT is set to launch, and on paper, it looks like a solid move. A $349 graphics card with 16GB of VRAM? Not bad. That’s more memory than some RTX 4070 cards. Sounds like AMD might finally be delivering some value again, right?  Well, yes and no.  Because right alongside that 16GB version, AMD is also releasing an 8GB version for $299. Same name, same chip, half the memory. And that’s where the internet lost it.  Déjà Vu: We’ve Seen This Trick Before If this sounds familiar, it’s because Nvidia pulled the same move with the RTX 4060 Ti.  They sold both 8GB and 16GB versions with the same branding, but a $100 price difference. The RTX 4060 Ti 8GB launched in May 2023, and the 16GB variant followed in July.  Source: Nvidia Gamers hated the confusion. Reviewers criticized the 8GB version’s lack of performance, especially in memory-heavy games, and the way Nvidia tried to sweep the difference under the rug.  Performance dipped significantly at 1440p, and stuttering was a problem even in some 1080p titles. The backlash was swift. Tech media slammed Nvidia for deceptive marketing, and buyers were left second-guessing which version they were getting.  We’ve seen this pattern before in Nvidia’s review restrictions around the RTX 5060, where early coverage was shaped by what reviewers were allowed to test – and what they weren’t.  It led to a mess of misinformation, bad value perceptions, and a very clear message: don’t confuse your customers. So naturally, AMD did it too.  It’s like watching two billion-dollar companies playing a game of ‘Who Can Confuse the Customer More.’ It’s not just about the money. It’s about trust, and AMD just dumped a bunch of it off a cliff.  Frank Azor Lights the Fuse on X The backlash started when AMD’s Director of Gaming Marketing, Frank Azor, took to X to defend the 8GB card.  He said that most gamers don’t need more than 8GB of VRAM and that the cheaper card still serves the mainstream crowd just fine.  It’s the same reasoning Nvidia used last year with the RTX 4060 Ti. That didn’t work then, and it isn’t working now.  Because when Steve from Hardware Unboxed sees a bad take like that, you know a flamethrower video is coming. And oh boy, did it come.  Hardware Unboxed Fires Back The backlash against AMD’s 8GB RX 9060 XT took off after a post from Hardware Unboxed on X called out the company’s defense of limited VRAM.  In response to AMD’s claim that most gamers don’t need more than 8GB of memory, Hardware Unboxed accused them of misleading buyers and building weaker products just to hit certain price points. The criticism gained traction fast. Tech YouTuber Vex picked up the story and added fuel to the fire by showing side-by-side gameplay comparisons.  In multiple games, the 8GB RX 9060 XT showed serious performance issues – stuttering, frame drops, and VRAM bottlenecks – while the 16GB version handled the same titles smoothly.  And yet, during the GPU’s official reveal, AMD only showed performance data for the 16GB card. There were no benchmarks for the 8GB version – not a single chart. That omission wasn’t lost on anyone. If AMD truly believed the 8GB model held up under modern gaming loads, they would have shown it. The silence speaks volumes.  Why This Actually Matters You might be thinking: ‘So what? Some games still run fine on 8GB. I only play Valorant.’ Sure. But the problem is bigger than that. Source: AMD Games are getting heavier. Even titles like Cyberpunk 2077, released in 2020, can eat up more than 8GB of VRAM. And with GTA 6 (still) on the horizon, do you really think game developers are going to keep optimizing for 8GB cards in 2025? That’s not how game development works. Developers target the most common setups, yes. But hardware also shapes software.  If everyone’s stuck with 8GB, games will be designed around that limit. That holds back progress for everyone.  It’s like trying to make a movie with a flip phone because some people still own one. Same Name, Different Game Another big issue is how these cards are named and sold.  The RX 9060 XT 16GB and RX 9060 XT 8GB are not clearly labeled as different products. They’re just two versions of the same GPU.  But that extra memory makes a huge difference.  In some games, the 8GB card performs dramatically worse. And yet, unless you know what to look for, you might walk into a store and buy the 8GB version thinking you’re getting the same performance.  You’re not. You’re getting a watered-down version with the same name and a silent asterisk. This isn’t just AMD’s Problem Nvidia started this mess with the 4060 Ti naming confusion. AMD just saw the outrage and decided to walk straight into the same buzzsaw.  It’s hard not to feel like both companies are treating consumers like they’re too dumb to notice. Spoiler: they noticed. And this whole ‘VRAM doesn’t matter’ argument? It’s already been debunked by dozens of reviewers.  If you’re spending over $300 on a graphics card in 2025, it needs to last more than a year or two. 8GB cards are already struggling. Buying one now is like buying a smartphone in 2025 with 64GB of storage. Sure, it works. Until it doesn’t. Steam Data Doesn’t Help AMD’s Case AMD and Nvidia both love to point at the Steam Hardware Survey. They say, ‘See? Most people still play at 1080p.’ And that’s true – for now. Source: Nvidia But what they leave out is that 1440p gaming is growing fast. More gamers are upgrading their setups because 1440p monitors are getting a lot more affordable.  Take the Pixio PXC277 Advanced, for instance – a 27-inch curved 1440p monitor with a 165Hz refresh rate and 1ms response time, all for $219.99. A few years ago, a screen like that would’ve cost you double. Now it’s entry-level. Gamers are ready to step up their experience. The only thing holding them back is GPU hardware that’s still stuck in 2020.  Planned Obsolescence in Disguise Here’s the worst part. Companies know full well that 8GB won’t cut it in 2026.  But they still sell it, knowing many gamers will only find out when it’s too late – when the stutters kick in, the textures disappear, or the next big title becomes unplayable. It’s planned obsolescence disguised as ‘choice.’ And while it’s great to have options at different price points, it should be clear which option is built to last – and which one is built to frustrate.  So, Is AMD Actually Screwed?  Not right now. In fact, they’re playing the game better than they used to.  They’ve learned from past pricing disasters and figured out how to get better launch-day headlines – even if it means faking the MSRP and letting street prices run wild.  But this kind of marketing comes at a cost. If AMD keeps making decisions that prioritize short-term wins over long-term trust, they’ll lose the very crowd that once rooted for them.  We don’t need two Nvidias. We need AMD to be different – to be better.  One Name, Two Very Different Cards The RX 9060 XT 16GB might be a good deal. But it’s being overshadowed by the 8GB version’s drama. And the longer AMD keeps playing games with memory and naming, the more it chips away at its hard-earned goodwill.  This whole mess could’ve been avoided with one simple move: name the 8GB card something else. Call it the RX 9055. Call it Lite or whatever. Just don’t make it look like the same card when it isn’t.  Until then, buyers beware. There’s more going on behind the box art than meets the eye.  Anya Zhukova is an in-house tech and crypto writer at Techreport with 10 years of hands-on experience covering cybersecurity, consumer tech, digital privacy, and blockchain. She’s known for turning complex topics into clear, useful advice that regular people can actually understand and use.  Her work has been featured in top-tier digital publications including MakeUseOf, Online Tech Tips, Help Desk Geek, Switching to Mac, and Make Tech Easier. Whether she’s writing about the latest privacy tools or reviewing a new laptop, her goal is always the same: help readers feel confident and in control of the tech they use every day.  Anya holds a BA in English Philology and Translation from Tula State Pedagogical University and also studied Mass Media and Journalism at Minnesota State University, Mankato. That mix of language, media, and tech has given her a unique lens to look at how technology shapes our daily lives.  Over the years, she’s also taken courses and done research in data privacy, digital security, and ethical writing – skills she uses when tackling sensitive topics like PC hardware, system vulnerabilities, and crypto security.  Anya worked directly with brands like Framework, Insta360, Redmagic, Inmotion, Secretlab, Kodak, and Anker, reviewing their products in real-life scenarios. Her testing process involves real-world use cases – whether it's stress-testing laptops for creative workloads, reviewing the battery performance of mobile gaming phones, or evaluating the long-term ergonomics of furniture designed for hybrid workspaces.  In the world of crypto, Anya covers everything from beginner guides to deep dives into hardware wallets, DeFi protocols, and Web3 tools. She helps readers understand how to use multisig wallets, keep their assets safe, and choose the right platforms for their needs.  Her writing often touches on financial freedom and privacy – two things she strongly believes should be in everyone’s hands. Outside of writing, Anya contributes to editorial style guides focused on privacy and inclusivity, and she mentors newer tech writers on how to build subject matter expertise and write responsibly.  She sticks to high editorial standards, only recommends products she’s personally tested, and always aims to give readers the full picture.  You can find her on LinkedIn, where she shares more about her work and projects.  Key Areas of Expertise: Consumer Tech (laptops, phones, wearables, etc.) Cybersecurity and Digital Privacy PC/PC Hardware Blockchain, Crypto Wallets, and DeFi In-Depth Product Reviews and Buying Guides Whether she’s reviewing a new wallet or benchmarking a PC build, Anya brings curiosity, care, and a strong sense of responsibility to everything she writes. Her mission? To make the digital world a little easier – and safer – for everyone.  View all articles by Anya Zhukova Our editorial process The Tech Report editorial policy is centered on providing helpful, accurate content that offers real value to our readers. We only work with experienced writers who have specific knowledge in the topics they cover, including latest developments in technology, online privacy, cryptocurrencies, software, and more. Our editorial policy ensures that each topic is researched and curated by our in-house editors. We maintain rigorous journalistic standards, and every article is 100% written by real authors.

Nintendo has predicted it will sell 15 million Switch 2s in its current financial year. Analysts think the number is conservative; Nintendo says the price of the Switch 2 is what’s holding back that estimate. But what does that mean, really? Is it a reasonable target? If Nintendo hits it, does that guarantee the Switch 2 will be a massive hit? If a console sells out at launch, what does that tell us? Is there still a chance of a Wii U-style flop? It’s impossible to know for sure, but a look at all of Nintendo’s past console launches can provide some clues. I’ve dug deep into past Nintendo sales figures to determine which Nintendo consoles enjoyed the best launches. To get a more reliable picture than that initial, almost inevitable sellout, I’ve defined the launch as the console’s first year (or rather, first four financial quarters) on the market. Note that Nintendo only started reporting quarterly sales for its systems in the mid-2000s, and the earliest data is annual at best and hard to come by, so some of these numbers are approximate. Note, too, that older systems had staggered launches across the three major markets (Japan, North America, and Europe), sometimes over several years, slowing down their potential sales. Still, there are some surprising results here that put Nintendo’s 15 million forecast for Switch 2 in context. Selling that many units would definitely not be bad news — but it doesn’t indicate a slam-dunk, either. 1. Game Boy Advance First four quarters: approx. 18.1 million Lifetime: 81.51 million (5th overall) Release: March-June 2001 2. 3DS First four quarters: 15.03 million Lifetime: 75.94 million (6th) Release: February-March 2011 There’s a clear pattern to Nintendo’s two fastest sellers; they were successors to massive hits (the Game Boy and DS) in the handheld market, where Nintendo enjoyed total dominance. Nintendo was so bullish about Game Boy Advance, following the decade-long reign of the Game Boy format, that it forecast an astonishing 24 million sales in its first year, while 3DS followed Nintendo’s biggest seller ever. Both sold well, but neither quite lived up to their forebears. 3. Switch First four quarters: 14.86 million Lifetime: 152.12 million (2nd, for now) Release: March 2017 4. Wii First four quarters: 13.17 million Lifetime: 101.63 million (4th) Release: November-December 2006 The Switch and the Wii are the only Nintendo consoles with sales over 10 million in the first year and over 100 million in their lifetimes. Quarter for quarter, they both sold incredibly consistently over time. This is surely what Nintendo would like all its hardware launches to look like, and what it’s hoping for with the Switch 2. 5. DS First four quarters: 8.83 million Lifetime: 154.02m (1st, for now) Release: November 2004-March 2005 It’s a surprise DS didn’t launch more strongly, considering the runaway early sales of its handheld predecessor, Game Boy Advance. But its launch games weren’t the best and it took a while for the console’s strange design to find its eventual massive casual audience. In its fifth quarter — holiday 2005 — it suddenly took off. 6. GameCube First four quarters: approx. 6.7 million Lifetime: 21.74 million (10th) Release: Sep 2001-May 2002 7. Nintendo 64 First four quarters: 5.80 million Lifetime: 32.93 million (9th) Release: June 1996-March 1997 The two Nintendo home consoles made during PlayStation’s ascendency enjoyed reasonably strong launches but petered out due to a lack of software support. GameCube sold 30% of its lifetime total in its first year on sale — an unfortunate achievement that beats even Wii U’s dismal ratio. 8. Game Boy First four quarters: 3.93 million Lifetime: 118.69 million (3rd) Release: April 1989-September 1990 The legend of Game Boy is that it was an instant smash, thanks to the high-stakes acquisition of Tetris. But while its launch was healthy for the time, it didn’t begin to indicate what Nintendo’s defining handheld would achieve — sales in the first year (before it reached Europe) were just 3% of what it would go on to sell across Game Boy and Game Boy Color. 9. Wii U First four quarters: 3.91 million Lifetime: 13.56 million (11th) Release: November-December 2012 Nintendo’s most recent (but not its worst) flop actually started quite strongly, shifting over 3 million units in its first quarter — but then suffered a catastrophic drop-off, selling less than 900,000 worldwide during the rest of its first year on sale. Looking past the initial holiday sell-out, the signs of the disaster to come were clear. 10. Super Famicom/SNES First four quarters: approx. 2.9 million (mostly Japan) Lifetime: 49.1 million (8th) Release: November 1990-June 1992  11. Famicom/NES Launch sales: 2.5 million by end of 1984 (Japan only) Lifetime: 61.91 million (7th) Release: July 1983 to 1987 and later Both Nintendo’s early home consoles launched strongly in Japan but took a long time to take off in the West. The NES didn’t fully launch in the U.S. and Europe until 1986, amid caution after the 1980s video game crash, while the SNES was beaten to market in the West by the Sega Genesis and suffered as a result.  12. Virtual Boy Launch and lifetime sales: 770,000 Release: July-August 1995 You can’t get a worse launch than being discontinued within a year of going on sale! — What does this mean for Switch 2’s launch? If Nintendo meets or exceeds its target of 15 million Switch 2s sold in its first financial year, it will rank among the top Nintendo console launches ever. A Wii U-style flop looks very unlikely, unless sales drop off sharply after its first few months. But the most telling comparisons here are Nintendo’s top two launches: Game Boy Advance and 3DS. Like Switch 2, they were both conservative, easy-to-understand sequels to huge sellers in a market sector Nintendo had total control of. And while both reached respectable lifetime totals, they got nowhere near the lifetime sales of their more innovative predecessors. Could this be Switch 2’s fate? Judging by Nintendo’s launch history, it’s possible — maybe even likely.

The big Leslie Benzies interview: MindsEye, Everywhere, and the double-edged sword of GTA How Build A Rocket Boy developed its debut project Feature by Samuel Roberts Editorial Director Published on May 30, 2025 As the producer behind the Grand Theft Auto games from GTA 3 through to GTA 5, as well as Red Dead Redemption and LA Noire, any project with Leslie Benzies' name on it is going to be a lightning rod for attention. MindsEye, the first game from Benzies' studio Build A Rocket Boy, is getting plenty of it – even if some of that attention has been less positive. MindsEye is a single-player third-person shooter with vehicle gameplay, set in a Las Vegas-style city called Redrock. It's a techno-thriller story about a former soldier called Jacob Diaz – but it's clear from visiting BARB in Edinburgh this week that the game is envisioned as a gateway into something much larger, both in the fiction of MindsEye, and for players who pick the game up. That includes a user-generated content platform called Build.MindsEye, where players on PC can create levels using relatively straightforward tools that incorporate any object in the game. When asked if third-person shooter levels or driving sections were the limits of the build side of MindsEye, the developers showed other examples of how they can be used, like massively increasing the proportions of a basketball, dropping it into the world, and functionally making an in-game version of Rocket League. Still, while MindsEye launches on June 10, 2025, for PC and consoles, many questions remain unanswered, including the future of its long-gestating Everywhere project. Benzies sat down with GamesIndustry.biz earlier this week to talk us through his vision for the game. This interview has been edited for brevity and clarity. Image credit: Austin Hargrave What's your grand vision for MindsEye? What will it be at launch, and where is it going in the future? MindsEye is one story in an epic universe. The other stories take place at different time periods, and [at] different locations in the universe. This story is Jacob Diaz's story. There are also other stories within MindsEye, so we tell the backstories of other characters Jacob will meet. That's the way we're going to fill out the universe over time – so when you travel around, all the stories will be connected by one overarching theme, and each story will have different mechanics. And we'll give these mechanics to players within the creator tools. What will happen with the game after launch? [The studio] will support the game through Play.MindsEye, with continuous new content. Some of the content, like races, are made just for fun. But [with] most of the content, we'll try and incorporate it into the story. So once you've played the big overarching ten-year plan, you'll have a very good idea of what this universe looks like. We have plans to add multiplayer, [and] we have plans to make a full open world. And of course, we've also got to look at what players are creating, and incorporate that into our plans. Given the ease of the tools, we think there's going to be a high percentage of players who will jump in and give it a pop, see how it feels. Hopefully some will create compelling content we can then promote and make that part of our plans to push to other players. Is it best to think of MindsEye as the first game in a series of games? Or one game as part of a larger experience? MindsEye sits bang in the middle of our story. So, we're going to go back 10,000 years, then we're going to go forward a certain amount of time. It's the relevant piece of the puzzle that will have players asking questions of what the bigger story is. We've intentionally not released footage of huge parts of the game, because we don't want to spoil anything for players. But this story does take some unusual twists. What's your vision for the multiplayer component of the game? I guess there's two sides to the answer. The dream from the building side is to allow players the opportunity to create their own multiplayer open world games with ease. So anyone could pick up the game, jump in, drive around, stop at a point where they see something of interest, build a little mission, jump back in the car, drive again, build another mission. Once you've built a couple of hundred of these, you've built your own open world game. So, that's the build side. From our side, we want to [create] a place where people can socialise, play together, and engage in the stories that we build. So, we do have plans next year to launch an open world multiplayer game that takes place a year after MindsEye finishes. In the interim, we also have an open world free roam game that spans from when MindsEye finishes to the launch of the open world multiplayer game. All of these stories interconnect in a fairly unique and original way, which I think players like these days. They like the complexity of deeper stories. You're selling the base game at launch, with a pass for upcoming content additions. Do you have a vision for how you're going to package future stories in the overarching MindsEye experience? It depends on the scale of the story. Some will be free, and some will be paid. After you left Rockstar Games, what came next? What led to you building the studio? I spent a few years looking into some other things: going [into] some property development. Using some of the games experience, we made a thing called VR-Chitect, which allowed you to build houses and view them in VR. I spent a lot of time in Los Angeles at this point, and this is when the droughts were very bad. I got into [making] these machines that would suck water out of the air. Still sitting in my back garden in Los Angeles is this big clunky machine, it works like an air-conditioning unit. It could suck up one thousand litres of water. So I got involved with that. But there's really nothing like making games. The different types of people – the lawyers, the accountants, the programmers, the artists, the dancers, the singers – that bunch of people in one big pot, all working together, and turning something from a piece of paper into [something on the] screen – that's where I get my excitement. Since I was a kid, that's what I've wanted to do. I thought, 'I better get back into making games' because nothing else was as much fun. What was the journey towards creating MindsEye as your first standalone release? Your first game's always your hardest. You have to build systems, you have to build the team. Everything is new. You don't really see a lot on the screen until way down the line, because you're building underlying systems, physics systems, the gameplay systems. It's a slow start, but what you end up with is an engine, and obviously we use Unreal, which provides a certain level of support and building. On top of that, we've got to build our own stuff. [Plus], we have to pack up everything we build and present it nicely for the creator tools. So it adds this extra layer of complexity to everything. But now, given where we are, the speed that we can iterate, we can very quickly place enemies, place vehicles, place puzzles, whatever, and get a feel for a game. We've now got a great, experienced team – a lot of talented guys in there. In the old days, you'd get a game, stick it on the shelf, and you'd wave goodbye. It's not like that anymore. You're continually fixing things. When you release a game, you've suddenly got, not a hundred testers, but hopefully millions of testers. You've got to continually fix, continually optimise, and especially with the tools that we've got, we want to continually create new content. So MindsEye is a standalone game, and Everywhere is not mentioned anywhere on the Steam page. But obviously there's a strong 'build' component to this game, which was part of the Everywhere pitch. What does this mean for Everywhere, and what was behind the decision to package the game this way? This is all part of a bigger story and ecosystem that we've got planned. Everywhere is going to show up again pretty soon. Everything we're working on, there's a story behind it – a big overarching story. So Everywhere will come back, and it fits into this story somewhere. I can't tell you [where], because it would be a spoiler. But that's going to reappear soon, and it will all be a part of the same product. "I'm not sure it would've been smart as a company to say, 'we are going to compete with the biggest game on the planet'" Leslie Benzies, Build A Rocket Boy In terms of the tools, the tool doesn't really care what world you're building in. It sits separately. So any game we create, it will naturally work on top of it. But we're big fans of keeping everything thematically connected, or connected through a narrative, and you'll see it. The bigger story will become obvious, once you've played through all of MindsEye. Then you might start to see how it all connects together, to the Everywhere world. Has the landscape for something like Everywhere, or the build component to MindsEye, changed as platforms like UEFN have taken off or Roblox has become so huge? It's great to see these tools being used by people. I build a lot with my son, and when he builds, I see the excitement he gets. It reminds me of when I was a kid with my Dragon 32 computer, managing to get a little character moving on the screen – that excitement of, 'wow, I did that'. Giving that to other people is massive. It's still very difficult to build in Roblox. For example, when my son wants to do it, I have to jump in. I used to be a programmer, and I struggle to build in there. When he wants to run around and scream with his friends he's in Roblox; when he wants to build he'll jump into Minecraft, because Minecraft is a much easier system to build within. And I think we sit somewhere in the middle: you can get very high quality, fun games, but they're very easy to build. I think we're at the infancy of this in video games. We're at the very beginning of it, and we're going to see way, way more of it. It doesn't necessarily have to be presenting it to your friends, or to an audience. I think the process of creating for a human being is fun in itself. MindsEye has been positioned as a linear game. You are best known for creating open world games. What was behind the decision to make MindsEye a more linear, narrative-driven experience? I think certain stories are more difficult to present to players in an open world setting. Open world gives you freedom – you don't necessarily want freedom to portray a story. For MindsEye, it's a very set time in a character, Jacob Diaz's, life. You pick up as Jacob when he arrives in Redrock, and then you leave Jacob at a certain point in the future. And so, it'd be very difficult for us to have an open world in there. It's horses for courses: it depends what you're doing. But for Jacob's story, it had to be a linear game. Having said that, there are open world experiences in there, and we can build them through Build.MindsEye. There is a free roam open world mode, where you play [as] a different character and you see his time, from the end of MindsEye, to the point of our next big planned launch. Again, they're all connected through a narrative, and we really want to show the universe, show the stories that have taken place in the universe, the characters in that universe, and see how they've experienced the same experience but from different viewpoints. "The dream from the building side is to allow players the opportunity to create their own multiplayer open world games with ease" Leslie Benzies, Build A Rocket Boy Was there ever a discussion about creating a more traditional GTA competitor? In design, you look at a lot of different options. I'm not sure it would've been smart as a company to say, 'we are going to compete with the biggest game on the planet'. I'm not sure that would be the best business decision to make. We went through a bunch of different designs, and to tell our story, this is what we landed on. MindsEye is priced more like a game from a decade ago at $60, and it'll take around 20 hours to finish. Can you talk about how you settled on the game's length and scope, and how you made that decision around price? So you've got the MindsEye campaign, and yes, it'll be about 20ish hours. But you do have all this other side content: there's going to be this continuous stream of content. These days, there are so many different options for people. It's not just games: there's streaming TV, so many good shows out there. I don't think you can have filler content in games. I think people want the meat, and they want the potatoes. We've tried to make as much meat as we can, if that makes sense. I think that's a good length for a game. What you also find through data, is that [with] big games, people don't play them all. The majority of people – 60% or 70% of people – don't actually play games to the end. So when you're making something, I would prefer – I'm sure the team would say the same – [that] you had the whole experience from start to finish, and not create this 200-hour game. Create something that is finishable, but have some side things that will fill out the universe. A lot of the side missions on the play side of MindsEye do fill out the characters' back stories, or do fill out what was happening in the world. On price: the world's in a funny place. People are worried about the price of eggs. So value for money, I think people appreciate that when times are difficult. I was curious why you waited until quite late in the day to reveal the build element of the game, only because it seemed you were being quite church and state with how MindsEye is releasing versus what Everywhere is. So in general, we believe – and again, it goes back to the amount of information, the amount of options people have these days – I don't think you can have extended marketing times. It's very expensive, we're a start-up. I think you lose interest from people. There are so many things for people to do, that if you extend it, you're not punching through to the place you need to be. I've seen other games, nine years before launch, it's getting talked about. I'm not sure that's the way of the world these days. You'll see there are games that never go to market: the day of launch was the marketing campaign, and it worked very well. So I think we tried to compress ours down for that reason. On the MindsEye.Play [continuous content] part of it, yeah, maybe we should've got that out there sooner, but it is a nice little surprise to give players. That's the thing with marketing – you never know what's the right or wrong way to do it, you've got to go with your gut, your senses, and test it. Being who you are, it brings a certain level of expectation and attention. Do you find it a double-edged sword, launching a new studio and launching a new game, with your background? Yes. There's always comparisons, and I think that's how humans work. As kids, we're taught to put a triangle into a triangular hole, and a square into a square hole. I think we do that for the rest of our lives, and we like to describe something new as 'it's X plus Y, with a bit of Z in there'. It makes things easy for us. It's maybe humans optimising the way we communicate. So there are comparisons. It serves us well in some ways, it doesn't serve us well in others. Dave Grohl said it well when he formed the Foo Fighters: nobody's interested in the Foo Fighters, all they were interested in was Nirvana. The guys have built something very cool, and I just hope people can see it for what it's trying to be.

The Fearsome Megalodon Ate Basically Whatever It Wanted to Reach Its Daily 100,000-Calorie Need, Study Suggests Scientists previously assumed the giant, prehistoric sharks mostly feasted on whales, but it turns out they probably weren’t so picky An artistic reconstruction of the extinct megalodon. Scientists' ideas about how the megalodon looked are based on its fossilized teeth. Hugo Saláis via Wikimedia Commons under CC BY 4.0 Between 3 million and 20 million years ago, the largest predatory fish ever known hunted in Earth’s oceans. Called theOtodus megalodon), this giant shark grew up to 79 feet long, had teeth the size of human hands and could bite with the strength of an industrial hydraulic press. But what scientists know about the extinct creature has been almost entirely determined from fossil teeth—since paleontologists have yet to discover a complete megalodon, and the animals’ cartilaginous skeletons don’t preserve well. Now, new research on the mineral content of their teeth suggests megalodons ate pretty much whatever they wanted. Until recently, scientists assumed that megalodons satisfied their estimated 100,000-calorie daily needs by mostly eating whales. A study published Monday in the journal Earth and Planetary Science Letters, however, suggests the prehistoric shark had a much more diverse diet than previously thought—akin to the great white shark’s “if it moves, it’s food” hunting strategy of today, writes Vice’s Ashley Fike. Jeremy McCormack with a fossilized megalodon tooth. Uwe Dettmar for Goethe University An international team of researchers reached this conclusion after analyzing the ratio of different variants, called isotopes, of the mineral zinc in 18-million-year-old megalodon teeth. Animals absorb zinc only through food, so this could offer a hint to their diets. Muscles and organs absorb more of the isotope zinc-64 than zinc-66, meaning that the higher up the food chain an animal is—or the more meat and fish it eats—the less zinc-66 it absorbs, and its ratio of zinc-66 to zinc-64 is lower, in turn. “Since we don’t know how the ratio of the two zinc isotopes at the bottom of the food pyramid was at that time, we compared the teeth of various prehistoric and extant shark species with each other and with other animal species. This enabled us to gain an impression of predator-prey relationships 18 million years ago,” Jeremy McCormack, a scientist from Goethe University Frankfurt and lead author of the study, says in a statement. Unsurprisingly, the isotope ratios in the teeth put the megalodon at the top of the food chain, alongside close shark relatives such as Otodus chubutensis. At the same time, however, the scientists noticed there wasn’t a huge difference between the megalodon and the lower-tiered animals, suggesting the sharks feasted on creatures from all rungs of the ladder. “They were not concentrating on certain prey types, but they must have fed throughout the food web, on many different species,” McCormack tells CNN’s Jacopo Prisco. “While certainly this was a fierce apex predator, and no one else would probably prey on an adult megalodon, it’s clear that they themselves could potentially feed on almost everything else that swam around.” The results also indicate that megalodon populations living in different habitats had slightly contrasting diets, potentially because of differing prey availability. More broadly, the study invites comparisons between the megalodon and its iconic extant relative, the great white shark. These comparisons, however, may have previously led to some overreaching assumptions. “Previous studies simply assumed that megalodon must have looked like a gigantic version of the modern great white shark without any evidence,” Kenshu Shimada, a vertebrate paleontologist at DePaul University and co-author of the new study, told National Geographic’s Jason Bittel back in March. He and colleagues had just published a different paper that reassessed the prehistoric shark’s size, suggesting that it had a more slender body than its smaller, modern cousin. The new study thus joins a host of research challenging widely held ideas about megalodons and their relatives, says Alberto Collareta, a paleontologist at the University of Pisa in Italy who was not involved in the research, to CNN. “These have led us to abandon traditional reconstruction of the megatooth sharks as ‘inflated’ versions of the modern white shark. We now know that the megalodon was something else—in terms of size, shape and ancestry, and of biology, too,” he adds. In fact, with both species eating generalist diets, great white sharks might have outcompeted megalodons for food and ultimately played a role in their demise. “Even ‘supercarnivores’ are not immune to extinction,” Shimada says in the statement. Get the latest stories in your inbox every weekday.

Transcript [MUSIC]    [BOOK PASSAGE]  PETER LEE: “In American primary care, the missing workforce is stunning in magnitude, the shortfall estimated to reach up to 48,000 doctors within the next dozen years. China and other countries with aging populations can expect drastic shortfalls, as well. Just last month, I asked a respected colleague retiring from primary care who he would recommend as a replacement; he told me bluntly that, other than expensive concierge care practices, he could not think of anyone, even for himself. This mismatch between need and supply will only grow, and the US is far from alone among developed countries in facing it.” [END OF BOOK PASSAGE]    [THEME MUSIC]    This is The AI Revolution in Medicine, Revisited. I’m your host, Peter Lee.    Shortly after OpenAI’s GPT-4 was publicly released, Carey Goldberg, Dr. Zak Kohane, and I published The AI Revolution in Medicine to help educate the world of healthcare and medical research about the transformative impact this new generative AI technology could have. But because we wrote the book when GPT-4 was still a secret, we had to speculate. Now, two years later, what did we get right, and what did we get wrong?     In this series, we’ll talk to clinicians, patients, hospital administrators, and others to understand the reality of AI in the field and where we go from here.      [THEME MUSIC FADES] The book passage I read at the top is from “Chapter 4: Trust but Verify,” which was written by Zak. You know, it’s no secret that in the US and elsewhere shortages in medical staff and the rise of clinician burnout are affecting the quality of patient care for the worse. In our book, we predicted that generative AI would be something that might help address these issues. So in this episode, we’ll delve into how individual performance gains that our previous guests have described might affect the healthcare workforce as a whole, and on the patient side, we’ll look into the influence of generative AI on the consumerization of healthcare. Now, since all of this consumes such a huge fraction of the overall economy, we’ll also get into what a general-purpose technology as disruptive as generative AI might mean in the context of labor markets and beyond.   To help us do that, I’m pleased to welcome Ethan Mollick and Azeem Azhar. Ethan Mollick is the Ralph J. Roberts Distinguished Faculty Scholar, a Rowan Fellow, and an associate professor at the Wharton School of the University of Pennsylvania. His research into the effects of AI on work, entrepreneurship, and education is applied by organizations around the world, leading him to be named one of Time magazine’s most influential people in AI for 2024. He’s also the author of the New York Times best-selling book Co-Intelligence. Azeem Azhar is an author, founder, investor, and one of the most thoughtful and influential voices on the interplay between disruptive emerging technologies and business and society. In his best-selling book, The Exponential Age, and in his highly regarded newsletter and podcast, Exponential View, he explores how technologies like AI are reshaping everything from healthcare to geopolitics. Ethan and Azeem are two leading thinkers on the ways that disruptive technologies—and especially AI—affect our work, our jobs, our business enterprises, and whole industries. As economists, they are trying to work out whether we are in the midst of an economic revolution as profound as the shift from an agrarian to an industrial society. [TRANSITION MUSIC] Here is my interview with Ethan Mollick: LEE: Ethan, welcome. ETHAN MOLLICK: So happy to be here, thank you. LEE: I described you as a professor at Wharton, which I think most of the people who listen to this podcast series know of as an elite business school. So it might surprise some people that you study AI. And beyond that, you know, that I would seek you out to talk about AI in medicine. [LAUGHTER] So to get started, how and why did it happen that you’ve become one of the leading experts on AI? MOLLICK: It’s actually an interesting story. I’ve been AI-adjacent my whole career. When I was [getting] my PhD at MIT, I worked with Marvin Minsky (opens in new tab) and the MIT [Massachusetts Institute of Technology] Media Labs AI group. But I was never the technical AI guy. I was the person who was trying to explain AI to everybody else who didn’t understand it. And then I became very interested in, how do you train and teach? And AI was always a part of that. I was building games for teaching, teaching tools that were used in hospitals and elsewhere, simulations. So when LLMs burst into the scene, I had already been using them and had a good sense of what they could do. And between that and, kind of, being practically oriented and getting some of the first research projects underway, especially under education and AI and performance, I became sort of a go-to person in the field. And once you’re in a field where nobody knows what’s going on and we’re all making it up as we go along—I thought it’s funny that you led with the idea that you have a couple of months head start for GPT-4, right. Like that’s all we have at this point, is a few months’ head start. [LAUGHTER] So being a few months ahead is good enough to be an expert at this point. Whether it should be or not is a different question. LEE: Well, if I understand correctly, leading AI companies like OpenAI, Anthropic, and others have now sought you out as someone who should get early access to really start to do early assessments and gauge early reactions. How has that been? MOLLICK: So, I mean, I think the bigger picture is less about me than about two things that tells us about the state of AI right now. One, nobody really knows what’s going on, right. So in a lot of ways, if it wasn’t for your work, Peter, like, I don’t think people would be thinking about medicine as much because these systems weren’t built for medicine. They weren’t built to change education. They weren’t built to write memos. They, like, they weren’t built to do any of these things. They weren’t really built to do anything in particular. It turns out they’re just good at many things. And to the extent that the labs work on them, they care about their coding ability above everything else and maybe math and science secondarily. They don’t think about the fact that it expresses high empathy. They don’t think about its accuracy and diagnosis or where it’s inaccurate. They don’t think about how it’s changing education forever. So one part of this is the fact that they go to my Twitter feed or ask me for advice is an indicator of where they are, too, which is they’re not thinking about this. And the fact that a few months’ head start continues to give you a lead tells you that we are at the very cutting edge. These labs aren’t sitting on projects for two years and then releasing them. Months after a project is complete or sooner, it’s out the door. Like, there’s very little delay. So we’re kind of all in the same boat here, which is a very unusual space for a new technology. LEE: And I, you know, explained that you’re at Wharton. Are you an odd fit as a faculty member at Wharton, or is this a trend now even in business schools that AI experts are becoming key members of the faculty? MOLLICK: I mean, it’s a little of both, right. It’s faculty, so everybody does everything. I’m a professor of innovation-entrepreneurship. I’ve launched startups before and working on that and education means I think about, how do organizations redesign themselves? How do they take advantage of these kinds of problems? So medicine’s always been very central to that, right. A lot of people in my MBA class have been MDs either switching, you know, careers or else looking to advance from being sort of individual contributors to running teams. So I don’t think that’s that bad a fit. But I also think this is general-purpose technology; it’s going to touch everything. The focus on this is medicine, but Microsoft does far more than medicine, right. It’s … there’s transformation happening in literally every field, in every country. This is a widespread effect. So I don’t think we should be surprised that business schools matter on this because we care about management. There’s a long tradition of management and medicine going together. There’s actually a great academic paper that shows that teaching hospitals that also have MBA programs associated with them have higher management scores and perform better (opens in new tab). So I think that these are not as foreign concepts, especially as medicine continues to get more complicated. LEE: Yeah. Well, in fact, I want to dive a little deeper on these issues of management, of entrepreneurship, um, education. But before doing that, if I could just stay focused on you. There is always something interesting to hear from people about their first encounters with AI. And throughout this entire series, I’ve been doing that both pre-generative AI and post-generative AI. So you, sort of, hinted at the pre-generative AI. You were in Minsky’s lab. Can you say a little bit more about that early encounter? And then tell us about your first encounters with generative AI. MOLLICK: Yeah. Those are great questions. So first of all, when I was at the media lab, that was pre-the current boom in sort of, you know, even in the old-school machine learning kind of space. So there was a lot of potential directions to head in. While I was there, there were projects underway, for example, to record every interaction small children had. One of the professors was recording everything their baby interacted with in the hope that maybe that would give them a hint about how to build an AI system. There was a bunch of projects underway that were about labeling every concept and how they relate to other concepts. So, like, it was very much Wild West of, like, how do we make an AI work—which has been this repeated problem in AI, which is, what is this thing? The fact that it was just like brute force over the corpus of all human knowledge turns out to be a little bit of like a, you know, it’s a miracle and a little bit of a disappointment in some ways [LAUGHTER] compared to how elaborate some of this was. So, you know, I think that, that was sort of my first encounters in sort of the intellectual way. The generative AI encounters actually started with the original, sort of, GPT-3, or, you know, earlier versions. And it was actually game-based. So I played games like AI Dungeon. And as an educator, I realized, oh my gosh, this stuff could write essays at a fourth-grade level. That’s really going to change the way, like, middle school works, was my thinking at the time. And I was posting about that back in, you know, 2021 that this is a big deal. But I think everybody was taken surprise, including the AI companies themselves, by, you know, ChatGPT, by GPT-3.5. The difference in degree turned out to be a difference in kind. LEE: Yeah, you know, if I think back, even with GPT-3, and certainly this was the case with GPT-2, it was, at least, you know, from where I was sitting, it was hard to get people to really take this seriously and pay attention. MOLLICK: Yes. LEE: You know, it’s remarkable. Within Microsoft, I think a turning point was the use of GPT-3 to do code completions. And that was actually productized as GitHub Copilot (opens in new tab), the very first version. That, I think, is where there was widespread belief. But, you know, in a way, I think there is, even for me early on, a sense of denial and skepticism. Did you have those initially at any point? MOLLICK: Yeah, I mean, it still happens today, right. Like, this is a weird technology. You know, the original denial and skepticism was, I couldn’t see where this was going. It didn’t seem like a miracle because, you know, of course computers can complete code for you. Like, what else are they supposed to do? Of course, computers can give you answers to questions and write fun things. So there’s difference of moving into a world of generative AI. I think a lot of people just thought that’s what computers could do. So it made the conversations a little weird. But even today, faced with these, you know, with very strong reasoner models that operate at the level of PhD students, I think a lot of people have issues with it, right. I mean, first of all, they seem intuitive to use, but they’re not always intuitive to use because the first use case that everyone puts AI to, it fails at because they use it like Google or some other use case. And then it’s genuinely upsetting in a lot of ways. I think, you know, I write in my book about the idea of three sleepless nights. That hasn’t changed. Like, you have to have an intellectual crisis to some extent, you know, and I think people do a lot to avoid having that existential angst of like, “Oh my god, what does it mean that a machine could think—apparently think—like a person?” So, I mean, I see resistance now. I saw resistance then. And then on top of all of that, there’s the fact that the curve of the technology is quite great. I mean, the price of GPT-4 level intelligence from, you know, when it was released has dropped 99.97% at this point, right. LEE: Yes. Mm-hmm. MOLLICK: I mean, I could run a GPT-4 class system basically on my phone. Microsoft’s releasing things that can almost run on like, you know, like it fits in almost no space, that are almost as good as the original GPT-4 models. I mean, I don’t think people have a sense of how fast the trajectory is moving either. LEE: Yeah, you know, there’s something that I think about often. There is this existential dread, or will this technology replace me? But I think the first people to feel that are researchers—people encountering this for the first time. You know, if you were working, let’s say, in Bayesian reasoning or in traditional, let’s say, Gaussian mixture model based, you know, speech recognition, you do get this feeling, Oh, my god, this technology has just solved the problem that I’ve dedicated my life to. And there is this really difficult period where you have to cope with that. And I think this is going to be spreading, you know, in more and more walks of life. And so this … at what point does that sort of sense of dread hit you, if ever? MOLLICK: I mean, you know, it’s not even dread as much as like, you know, Tyler Cowen wrote that it’s impossible to not feel a little bit of sadness as you use these AI systems, too. Because, like, I was talking to a friend, just as the most minor example, and his talent that he was very proud of was he was very good at writing limericks for birthday cards. He’d write these limericks. Everyone was always amused by them. [LAUGHTER] And now, you know, GPT-4 and GPT-4.5, they made limericks obsolete. Like, anyone can write a good limerick, right. So this was a talent, and it was a little sad. Like, this thing that you cared about mattered. You know, as academics, we’re a little used to dead ends, right, and like, you know, some getting the lap. But the idea that entire fields are hitting that way. Like in medicine, there’s a lot of support systems that are now obsolete. And the question is how quickly you change that. In education, a lot of our techniques are obsolete. What do you do to change that? You know, it’s like the fact that this brute force technology is good enough to solve so many problems is weird, right. And it’s not just the end of, you know, of our research angles that matter, too. Like, for example, I ran this, you know, 14-person-plus, multimillion-dollar effort at Wharton to build these teaching simulations, and we’re very proud of them. It took years of work to build one. Now we’ve built a system that can build teaching simulations on demand by you talking to it with one team member. And, you know, you literally can create any simulation by having a discussion with the AI. I mean, you know, there’s a switch to a new form of excitement, but there is a little bit of like, this mattered to me, and, you know, now I have to change how I do things. I mean, adjustment happens. But if you haven’t had that displacement, I think that’s a good indicator that you haven’t really faced AI yet. LEE: Yeah, what’s so interesting just listening to you is you use words like sadness, and yet I can see the—and hear the—excitement in your voice and your body language. So, you know, that’s also kind of an interesting aspect of all of this.  MOLLICK: Yeah, I mean, I think there’s something on the other side, right. But, like, I can’t say that I haven’t had moments where like, ughhhh, but then there’s joy and basically like also, you know, freeing stuff up. I mean, I think about doctors or professors, right. These are jobs that bundle together lots of different tasks that you would never have put together, right. If you’re a doctor, you would never have expected the same person to be good at keeping up with the research and being a good diagnostician and being a good manager and being good with people and being good with hand skills. Like, who would ever want that kind of bundle? That’s not something you’re all good at, right. And a lot of our stress of our job comes from the fact that we suck at some of it. And so to the extent that AI steps in for that, you kind of feel bad about some of the stuff that it’s doing that you wanted to do. But it’s much more uplifting to be like, I don’t have to do this stuff I’m bad anymore, or I get the support to make myself good at it. And the stuff that I really care about, I can focus on more. Well, because we are at kind of a unique moment where whatever you’re best at, you’re still better than AI. And I think it’s an ongoing question about how long that lasts. But for right now, like you’re not going to say, OK, AI replaces me entirely in my job in medicine. It’s very unlikely. But you will say it replaces these 17 things I’m bad at, but I never liked that anyway. So it’s a period of both excitement and a little anxiety. LEE: Yeah, I’m going to want to get back to this question about in what ways AI may or may not replace doctors or some of what doctors and nurses and other clinicians do. But before that, let’s get into, I think, the real meat of this conversation. In previous episodes of this podcast, we talked to clinicians and healthcare administrators and technology developers that are very rapidly injecting AI today to do various forms of workforce automation, you know, automatically writing a clinical encounter note, automatically filling out a referral letter or request for prior authorization for some reimbursement to an insurance company. And so these sorts of things are intended not only to make things more efficient and lower costs but also to reduce various forms of drudgery, cognitive burden on frontline health workers. So how do you think about the impact of AI on that aspect of workforce, and, you know, what would you expect will happen over the next few years in terms of impact on efficiency and costs? MOLLICK: So I mean, this is a case where I think we’re facing the big bright problem in AI in a lot of ways, which is that this is … at the individual level, there’s lots of performance gains to be gained, right. The problem, though, is that we as individuals fit into systems, in medicine as much as anywhere else or more so, right. Which is that you could individually boost your performance, but it’s also about systems that fit along with this, right. So, you know, if you could automatically, you know, record an encounter, if you could automatically make notes, does that change what you should be expecting for notes or the value of those notes or what they’re for? How do we take what one person does and validate it across the organization and roll it out for everybody without making it a 10-year process that it feels like IT in medicine often is? Like, so we’re in this really interesting period where there’s incredible amounts of individual innovation in productivity and performance improvements in this field, like very high levels of it, but not necessarily seeing that same thing translate to organizational efficiency or gains. And one of my big concerns is seeing that happen. We’re seeing that in nonmedical problems, the same kind of thing, which is, you know, we’ve got research showing 20 and 40% performance improvements, like not uncommon to see those things. But then the organization doesn’t capture it; the system doesn’t capture it. Because the individuals are doing their own work and the systems don’t have the ability to, kind of, learn or adapt as a result. LEE: You know, where are those productivity gains going, then, when you get to the organizational level? MOLLICK: Well, they’re dying for a few reasons. One is, there’s a tendency for individual contributors to underestimate the power of management, right. Practices associated with good management increase happiness, decrease, you know, issues, increase success rates. In the same way, about 40%, as far as we can tell, of the US advantage over other companies, of US firms, has to do with management ability. Like, management is a big deal. Organizing is a big deal. Thinking about how you coordinate is a big deal. At the individual level, when things get stuck there, right, you can’t start bringing them up to how systems work together. It becomes, How do I deal with a doctor that has a 60% performance improvement? We really only have one thing in our playbook for doing that right now, which is, OK, we could fire 40% of the other doctors and still have a performance gain, which is not the answer you want to see happen. So because of that, people are hiding their use. They’re actually hiding their use for lots of reasons. And it’s a weird case because the people who are able to figure out best how to use these systems, for a lot of use cases, they’re actually clinicians themselves because they’re experimenting all the time. Like, they have to take those encounter notes. And if they figure out a better way to do it, they figure that out. You don’t want to wait for, you know, a med tech company to figure that out and then sell that back to you when it can be done by the physicians themselves. So we’re just not used to a period where everybody’s innovating and where the management structure isn’t in place to take advantage of that. And so we’re seeing things stalled at the individual level, and people are often, especially in risk-averse organizations or organizations where there’s lots of regulatory hurdles, people are so afraid of the regulatory piece that they don’t even bother trying to make change. LEE: If you are, you know, the leader of a hospital or a clinic or a whole health system, how should you approach this? You know, how should you be trying to extract positive success out of AI? MOLLICK: So I think that you need to embrace the right kind of risk, right. We don’t want to put risk on our patients … like, we don’t want to put uninformed risk. But innovation involves risk to how organizations operate. They involve change. So I think part of this is embracing the idea that R&D has to happen in organizations again. What’s happened over the last 20 years or so has been organizations giving that up. Partially, that’s a trend to focus on what you’re good at and not try and do this other stuff. Partially, it’s because it’s outsourced now to software companies that, like, Salesforce tells you how to organize your sales team. Workforce tells you how to organize your organization. Consultants come in and will tell you how to make change based on the average of what other people are doing in your field. So companies and organizations and hospital systems have all started to give up their ability to create their own organizational change. And when I talk to organizations, I often say they have to have two approaches. They have to think about the crowd and the lab. So the crowd is the idea of how to empower clinicians and administrators and supporter networks to start using AI and experimenting in ethical, legal ways and then sharing that information with each other. And the lab is, how are we doing R&D about the approach of how to [get] AI to work, not just in direct patient care, right. But also fundamentally, like, what paperwork can you cut out? How can we better explain procedures? Like, what management role can this fill? And we need to be doing active experimentation on that. We can’t just wait for, you know, Microsoft to solve the problems. It has to be at the level of the organizations themselves. LEE: So let’s shift a little bit to the patient. You know, one of the things that we see, and I think everyone is seeing, is that people are turning to chatbots, like ChatGPT, actually to seek healthcare information for, you know, their own health or the health of their loved ones. And there was already, prior to all of this, a trend towards, let’s call it, consumerization of healthcare. So just in the business of healthcare delivery, do you think AI is going to hasten these kinds of trends, or from the consumer’s perspective, what … ? MOLLICK: I mean, absolutely, right. Like, all the early data that we have suggests that for most common medical problems, you should just consult AI, too, right. In fact, there is a real question to ask: at what point does it become unethical for doctors themselves to not ask for a second opinion from the AI because it’s cheap, right? You could overrule it or whatever you want, but like not asking seems foolish. I think the two places where there’s a burning almost, you know, moral imperative is … let’s say, you know, I’m in Philadelphia, I’m a professor, I have access to really good healthcare through the Hospital University of Pennsylvania system. I know doctors. You know, I’m lucky. I’m well connected. If, you know, something goes wrong, I have friends who I can talk to. I have specialists. I’m, you know, pretty well educated in this space. But for most people on the planet, they don’t have access to good medical care, they don’t have good health. It feels like it’s absolutely imperative to say when should you use AI and when not. Are there blind spots? What are those things? And I worry that, like, to me, that would be the crash project I’d be invoking because I’m doing the same thing in education, which is this system is not as good as being in a room with a great teacher who also uses AI to help you, but it’s better than not getting an, you know, to the level of education people get in many cases. Where should we be using it? How do we guide usage in the right way? Because the AI labs aren’t thinking about this. We have to. So, to me, there is a burning need here to understand this. And I worry that people will say, you know, everything that’s true—AI can hallucinate, AI can be biased. All of these things are absolutely true, but people are going to use it. The early indications are that it is quite useful. And unless we take the active role of saying, here’s when to use it, here’s when not to use it, we don’t have a right to say, don’t use this system. And I think, you know, we have to be exploring that. LEE: What do people need to understand about AI? And what should schools, universities, and so on be teaching? MOLLICK: Those are, kind of, two separate questions in lot of ways. I think a lot of people want to teach AI skills, and I will tell you, as somebody who works in this space a lot, there isn’t like an easy, sort of, AI skill, right. I could teach you prompt engineering in two to three classes, but every indication we have is that for most people under most circumstances, the value of prompting, you know, any one case is probably not that useful. A lot of the tricks are disappearing because the AI systems are just starting to use them themselves. So asking good questions, being a good manager, being a good thinker tend to be important, but like magic tricks around making, you know, the AI do something because you use the right phrase used to be something that was real but is rapidly disappearing. So I worry when people say teach AI skills. No one’s been able to articulate to me as somebody who knows AI very well and teaches classes on AI, what those AI skills that everyone should learn are, right. I mean, there’s value in learning a little bit how the models work. There’s a value in working with these systems. A lot of it’s just hands on keyboard kind of work. But, like, we don’t have an easy slam dunk “this is what you learn in the world of AI” because the systems are getting better, and as they get better, they get less sensitive to these prompting techniques. They get better prompting themselves. They solve problems spontaneously and start being agentic. So it’s a hard problem to ask about, like, what do you train someone on? I think getting people experience in hands-on-keyboards, getting them to … there’s like four things I could teach you about AI, and two of them are already starting to disappear. But, like, one is be direct. Like, tell the AI exactly what you want. That’s very helpful. Second, provide as much context as possible. That can include things like acting as a doctor, but also all the information you have. The third is give it step-by-step directions—that’s becoming less important. And the fourth is good and bad examples of the kind of output you want. Those four, that’s like, that’s it as far as the research telling you what to do, and the rest is building intuition. LEE: I’m really impressed that you didn’t give the answer, “Well, everyone should be teaching my book, Co-Intelligence.” [LAUGHS] MOLLICK: Oh, no, sorry! Everybody should be teaching my book Co-Intelligence. I apologize. [LAUGHTER] LEE: It’s good to chuckle about that, but actually, I can’t think of a better book, like, if you were to assign a textbook in any professional education space, I think Co-Intelligence would be number one on my list. Are there other things that you think are essential reading? MOLLICK: That’s a really good question. I think that a lot of things are evolving very quickly. I happen to, kind of, hit a sweet spot with Co-Intelligence to some degree because I talk about how I used it, and I was, sort of, an advanced user of these systems. So, like, it’s, sort of, like my Twitter feed, my online newsletter. I’m just trying to, kind of, in some ways, it’s about trying to make people aware of what these systems can do by just showing a lot, right. Rather than picking one thing, and, like, this is a general-purpose technology. Let’s use it for this. And, like, everybody gets a light bulb for a different reason. So more than reading, it is using, you know, and that can be Copilot or whatever your favorite tool is. But using it. Voice modes help a lot. In terms of readings, I mean, I think that there is a couple of good guides to understanding AI that were originally blog posts. I think Tim Lee has one called Understanding AI (opens in new tab), and it had a good overview … LEE: Yeah, that’s a great one. MOLLICK: … of that topic that I think explains how transformers work, which can give you some mental sense. I think [Andrej] Karpathy (opens in new tab) has some really nice videos of use that I would recommend. Like on the medical side, I think the book that you did, if you’re in medicine, you should read that. I think that that’s very valuable. But like all we can offer are hints in some ways. Like there isn’t … if you’re looking for the instruction manual, I think it can be very frustrating because it’s like you want the best practices and procedures laid out, and we cannot do that, right. That’s not how a system like this works. LEE: Yeah. MOLLICK: It’s not a person, but thinking about it like a person can be helpful, right. LEE: One of the things that has been sort of a fun project for me for the last few years is I have been a founding board member of a new medical school at Kaiser Permanente. And, you know, that medical school curriculum is being formed in this era. But it’s been perplexing to understand, you know, what this means for a medical school curriculum. And maybe even more perplexing for me, at least, is the accrediting bodies, which are extremely important in US medical schools; how accreditors should think about what’s necessary here. Besides the things that you’ve … the, kind of, four key ideas you mentioned, if you were talking to the board of directors of the LCME [Liaison Committee on Medical Education] accrediting body, what’s the one thing you would want them to really internalize? MOLLICK: This is both a fast-moving and vital area. This can’t be viewed like a usual change, which [is], “Let’s see how this works.” Because it’s, like, the things that make medical technologies hard to do, which is like unclear results, limited, you know, expensive use cases where it rolls out slowly. So one or two, you know, advanced medical facilities get access to, you know, proton beams or something else at multi-billion dollars of cost, and that takes a while to diffuse out. That’s not happening here. This is all happening at the same time, all at once. This is now … AI is part of medicine. I mean, there’s a minor point that I’d make that actually is a really important one, which is large language models, generative AI overall, work incredibly differently than other forms of AI. So the other worry I have with some of these accreditors is they blend together algorithmic forms of AI, which medicine has been trying for long time—decision support, algorithmic methods, like, medicine more so than other places has been thinking about those issues. Generative AI, even though it uses the same underlying techniques, is a completely different beast. So, like, even just take the most simple thing of algorithmic aversion, which is a well-understood problem in medicine, right. Which is, so you have a tool that could tell you as a radiologist, you know, the chance of this being cancer; you don’t like it, you overrule it, right. We don’t find algorithmic aversion happening with LLMs in the same way. People actually enjoy using them because it’s more like working with a person. The flaws are different. The approach is different. So you need to both view this as universal applicable today, which makes it urgent, but also as something that is not the same as your other form of AI, and your AI working group that is thinking about how to solve this problem is not the right people here. LEE: You know, I think the world has been trained because of the magic of web search to view computers as question-answering machines. Ask a question, get an answer. MOLLICK: Yes. Yes. LEE: Write a query, get results. And as I have interacted with medical professionals, you can see that medical professionals have that model of a machine in mind. And I think that’s partly, I think psychologically, why hallucination is so alarming. Because you have a mental model of a computer as a machine that has absolutely rock-solid perfect memory recall. But the thing that was so powerful in Co-Intelligence, and we tried to get at this in our book also, is that’s not the sweet spot. It’s this sort of deeper interaction, more of a collaboration. And I thought your use of the term Co-Intelligence really just even in the title of the book tried to capture this. When I think about education, it seems like that’s the first step, to get past this concept of a machine being just a question-answering machine. Do you have a reaction to that idea? MOLLICK: I think that’s very powerful. You know, we’ve been trained over so many years at both using computers but also in science fiction, right. Computers are about cold logic, right. They will give you the right answer, but if you ask it what love is, they explode, right. Like that’s the classic way you defeat the evil robot in Star Trek, right. “Love does not compute.” [LAUGHTER] Instead, we have a system that makes mistakes, is warm, beats doctors in empathy in almost every controlled study on the subject, right. Like, absolutely can outwrite you in a sonnet but will absolutely struggle with giving you the right answer every time. And I think our mental models are just broken for this. And I think you’re absolutely right. And that’s part of what I thought your book does get at really well is, like, this is a different thing. It’s also generally applicable. Again, the model in your head should be kind of like a person even though it isn’t, right. There’s a lot of warnings and caveats to it, but if you start from person, smart person you’re talking to, your mental model will be more accurate than smart machine, even though both are flawed examples, right. So it will make mistakes; it will make errors. The question is, what do you trust it on? What do you not trust it? As you get to know a model, you’ll get to understand, like, I totally don’t trust it for this, but I absolutely trust it for that, right. LEE: All right. So we’re getting to the end of the time we have together. And so I’d just like to get now into something a little bit more provocative. And I get the question all the time. You know, will AI replace doctors? In medicine and other advanced knowledge work, project out five to 10 years. What do think happens? MOLLICK: OK, so first of all, let’s acknowledge systems change much more slowly than individual use. You know, doctors are not individual actors; they’re part of systems, right. So not just the system of a patient who like may or may not want to talk to a machine instead of a person but also legal systems and administrative systems and systems that allocate labor and systems that train people. So, like, it’s hard to imagine that in five to 10 years medicine being so upended that even if AI was better than doctors at every single thing doctors do, that we’d actually see as radical a change in medicine as you might in other fields. I think you will see faster changes happen in consulting and law and, you know, coding, other spaces than medicine. But I do think that there is good reason to suspect that AI will outperform people while still having flaws, right. That’s the difference. We’re already seeing that for common medical questions in enough randomized controlled trials that, you know, best doctors beat AI, but the AI beats the mean doctor, right. Like, that’s just something we should acknowledge is happening at this point. Now, will that work in your specialty? No. Will that work with all the contingent social knowledge that you have in your space? Probably not. Like, these are vignettes, right. But, like, that’s kind of where things are. So let’s assume, right … you’re asking two questions. One is, how good will AI get? LEE: Yeah. MOLLICK: And we don’t know the answer to that question. I will tell you that your colleagues at Microsoft and increasingly the labs, the AI labs themselves, are all saying they think they’ll have a machine smarter than a human at every intellectual task in the next two to three years. If that doesn’t happen, that makes it easier to assume the future, but let’s just assume that that’s the case. I think medicine starts to change with the idea that people feel obligated to use this to help for everything. Your patients will be using it, and it will be your advisor and helper at the beginning phases, right. And I think that I expect people to be better at empathy. I expect better bedside manner. I expect management tasks to become easier. I think administrative burden might lighten if we handle this right way or much worse if we handle it badly. Diagnostic accuracy will increase, right. And then there’s a set of discovery pieces happening, too, right. One of the core goals of all the AI companies is to accelerate medical research. How does that happen and how does that affect us is a, kind of, unknown question. So I think clinicians are in both the eye of the storm and surrounded by it, right. Like, they can resist AI use for longer than most other fields, but everything around them is going to be affected by it. LEE: Well, Ethan, this has been really a fantastic conversation. And, you know, I think in contrast to all the other conversations we’ve had, this one gives especially the leaders in healthcare, you know, people actually trying to lead their organizations into the future, whether it’s in education or in delivery, a lot to think about. So I really appreciate you joining. MOLLICK: Thank you. [TRANSITION MUSIC]   I’m a computing researcher who works with people who are right in the middle of today’s bleeding-edge developments in AI. And because of that, I often lose sight of how to talk to a broader audience about what it’s all about. And so I think one of Ethan’s superpowers is that he has this knack for explaining complex topics in AI in a really accessible way, getting right to the most important points without making it so simple as to be useless. That’s why I rarely miss an opportunity to read up on his latest work. One of the first things I learned from Ethan is the intuition that you can, sort of, think of AI as a very knowledgeable intern. In other words, think of it as a persona that you can interact with, but you also need to be a manager for it and to always assess the work that it does. In our discussion, Ethan went further to stress that there is, because of that, a serious education gap. You know, over the last decade or two, we’ve all been trained, mainly by search engines, to think of computers as question-answering machines. In medicine, in fact, there’s a question-answering application that is really popular called UpToDate (opens in new tab). Doctors use it all the time. But generative AI systems like ChatGPT are different. There’s therefore a challenge in how to break out of the old-fashioned mindset of search to get the full value out of generative AI. The other big takeaway for me was that Ethan pointed out while it’s easy to see productivity gains from AI at the individual level, those same gains, at least today, don’t often translate automatically to organization-wide or system-wide gains. And one, of course, has to conclude that it takes more than just making individuals more productive; the whole system also has to adjust to the realities of AI. Here’s now my interview with Azeem Azhar: LEE: Azeem, welcome. AZEEM AZHAR: Peter, thank you so much for having me.  LEE: You know, I think you’re extremely well known in the world. But still, some of the listeners of this podcast series might not have encountered you before. And so one of the ways I like to ask people to introduce themselves is, how do you explain to your parents what you do every day? AZHAR: Well, I’m very lucky in that way because my mother was the person who got me into computers more than 40 years ago. And I still have that first computer, a ZX81 with a Z80 chip … LEE: Oh wow. AZHAR: … to this day. It sits in my study, all seven and a half thousand transistors and Bakelite plastic that it is. And my parents were both economists, and economics is deeply connected with technology in some sense. And I grew up in the late ’70s and the early ’80s. And that was a time of tremendous optimism around technology. It was space opera, science fiction, robots, and of course, the personal computer and, you know, Bill Gates and Steve Jobs. So that’s where I started. And so, in a way, my mother and my dad, who passed away a few years ago, had always known me as someone who was fiddling with computers but also thinking about economics and society. And so, in a way, it’s easier to explain to them because they’re the ones who nurtured the environment that allowed me to research technology and AI and think about what it means to firms and to the economy at large. LEE: I always like to understand the origin story. And what I mean by that is, you know, what was your first encounter with generative AI? And what was that like? What did you go through? AZHAR: The first real moment was when Midjourney and Stable Diffusion emerged in that summer of 2022. I’d been away on vacation, and I came back—and I’d been off grid, in fact—and the world had really changed. Now, I’d been aware of GPT-3 and GPT-2, which I played around with and with BERT, the original transformer paper about seven or eight years ago, but it was the moment where I could talk to my computer, and it could produce these images, and it could be refined in natural language that really made me think we’ve crossed into a new domain. We’ve gone from AI being highly discriminative to AI that’s able to explore the world in particular ways. And then it was a few months later that ChatGPT came out—November, the 30th. And I think it was the next day or the day after that I said to my team, everyone has to use this, and we have to meet every morning and discuss how we experimented the day before. And we did that for three or four months. And, you know, it was really clear to me in that interface at that point that, you know, we’d absolutely pass some kind of threshold. LEE: And who’s the we that you were experimenting with? AZHAR: So I have a team of four who support me. They’re mostly researchers of different types. I mean, it’s almost like one of those jokes. You know, I have a sociologist, an economist, and an astrophysicist. And, you know, they walk into the bar, [LAUGHTER] or they walk into our virtual team room, and we try to solve problems. LEE: Well, so let’s get now into brass tacks here. And I think I want to start maybe just with an exploration of the economics of all this and economic realities. Because I think in a lot of your work—for example, in your book—you look pretty deeply at how automation generally and AI specifically are transforming certain sectors like finance, manufacturing, and you have a really, kind of, insightful focus on what this means for productivity and which ways, you know, efficiencies are found.   And then you, sort of, balance that with risks, things that can and do go wrong. And so as you take that background and looking at all those other sectors, in what ways are the same patterns playing out or likely to play out in healthcare and medicine? AZHAR: I’m sure we will see really remarkable parallels but also new things going on. I mean, medicine has a particular quality compared to other sectors in the sense that it’s highly regulated, market structure is very different country to country, and it’s an incredibly broad field. I mean, just think about taking a Tylenol and going through laparoscopic surgery. Having an MRI and seeing a physio. I mean, this is all medicine. I mean, it’s hard to imagine a sector that is [LAUGHS] more broad than that. So I think we can start to break it down, and, you know, where we’re seeing things with generative AI will be that the, sort of, softest entry point, which is the medical scribing. And I’m sure many of us have been with clinicians who have a medical scribe running alongside—they’re all on Surface Pros I noticed, right? [LAUGHTER] They’re on the tablet computers, and they’re scribing away. And what that’s doing is, in the words of my friend Eric Topol, it’s giving the clinician time back (opens in new tab), right. They have time back from days that are extremely busy and, you know, full of administrative overload. So I think you can obviously do a great deal with reducing that overload. And within my team, we have a view, which is if you do something five times in a week, you should be writing an automation for it. And if you’re a doctor, you’re probably reviewing your notes, writing the prescriptions, and so on several times a day. So those are things that can clearly be automated, and the human can be in the loop. But I think there are so many other ways just within the clinic that things can help. So, one of my friends, my friend from my junior school—I’ve known him since I was 9—is an oncologist who’s also deeply into machine learning, and he’s in Cambridge in the UK. And he built with Microsoft Research a suite of imaging AI tools from his own discipline, which they then open sourced. So that’s another way that you have an impact, which is that you actually enable the, you know, generalist, specialist, polymath, whatever they are in health systems to be able to get this technology, to tune it to their requirements, to use it, to encourage some grassroots adoption in a system that’s often been very, very heavily centralized. LEE: Yeah. AZHAR: And then I think there are some other things that are going on that I find really, really exciting. So one is the consumerization of healthcare. So I have one of those sleep tracking rings, the Oura (opens in new tab). LEE: Yup. AZHAR: That is building a data stream that we’ll be able to apply more and more AI to. I mean, right now, it’s applying traditional, I suspect, machine learning, but you can imagine that as we start to get more data, we start to get more used to measuring ourselves, we create this sort of pot, a personal asset that we can turn AI to. And there’s still another category. And that other category is one of the completely novel ways in which we can enable patient care and patient pathway. And there’s a fantastic startup in the UK called Neko Health (opens in new tab), which, I mean, does physicals, MRI scans, and blood tests, and so on. It’s hard to imagine Neko existing without the sort of advanced data, machine learning, AI that we’ve seen emerge over the last decade. So, I mean, I think that there are so many ways in which the temperature is slowly being turned up to encourage a phase change within the healthcare sector. And last but not least, I do think that these tools can also be very, very supportive of a clinician’s life cycle. I think we, as patients, we’re a bit …  I don’t know if we’re as grateful as we should be for our clinicians who are putting in 90-hour weeks. [LAUGHTER] But you can imagine a world where AI is able to support not just the clinicians’ workload but also their sense of stress, their sense of burnout. So just in those five areas, Peter, I sort of imagine we could start to fundamentally transform over the course of many years, of course, the way in which people think about their health and their interactions with healthcare systems LEE: I love how you break that down. And I want to press on a couple of things. You also touched on the fact that medicine is, at least in most of the world, is a highly regulated industry. I guess finance is the same way, but they also feel different because the, like, finance sector has to be very responsive to consumers, and consumers are sensitive to, you know, an abundance of choice; they are sensitive to price. Is there something unique about medicine besides being regulated? AZHAR: I mean, there absolutely is. And in finance, as well, you have much clearer end states. So if you’re not in the consumer space, but you’re in the, you know, asset management space, you have to essentially deliver returns against the volatility or risk boundary, right. That’s what you have to go out and do. And I think if you’re in the consumer industry, you can come back to very, very clear measures, net promoter score being a very good example. In the case of medicine and healthcare, it is much more complicated because as far as the clinician is concerned, people are individuals, and we have our own parts and our own responses. If we didn’t, there would never be a need for a differential diagnosis. There’d never be a need for, you know, Let’s try azithromycin first, and then if that doesn’t work, we’ll go to vancomycin, or, you know, whatever it happens to be. You would just know. But ultimately, you know, people are quite different. The symptoms that they’re showing are quite different, and also their compliance is really, really different. I had a back problem that had to be dealt with by, you know, a physio and extremely boring exercises four times a week, but I was ruthless in complying, and my physio was incredibly surprised. He’d say well no one ever does this, and I said, well you know the thing is that I kind of just want to get this thing to go away. LEE: Yeah. AZHAR: And I think that that’s why medicine is and healthcare is so different and more complex. But I also think that’s why AI can be really, really helpful. I mean, we didn’t talk about, you know, AI in its ability to potentially do this, which is to extend the clinician’s presence throughout the week. LEE: Right. Yeah. AZHAR: The idea that maybe some part of what the clinician would do if you could talk to them on Wednesday, Thursday, and Friday could be delivered through an app or a chatbot just as a way of encouraging the compliance, which is often, especially with older patients, one reason why conditions, you know, linger on for longer. LEE: You know, just staying on the regulatory thing, as I’ve thought about this, the one regulated sector that I think seems to have some parallels to healthcare is energy delivery, energy distribution. Because like healthcare, as a consumer, I don’t have choice in who delivers electricity to my house. And even though I care about it being cheap or at least not being overcharged, I don’t have an abundance of choice. I can’t do price comparisons. And there’s something about that, just speaking as a consumer of both energy and a consumer of healthcare, that feels similar. Whereas other regulated industries, you know, somehow, as a consumer, I feel like I have a lot more direct influence and power. Does that make any sense to someone, you know, like you, who’s really much more expert in how economic systems work? AZHAR: I mean, in a sense, one part of that is very, very true. You have a limited panel of energy providers you can go to, and in the US, there may be places where you have no choice. I think the area where it’s slightly different is that as a consumer or a patient, you can actually make meaningful choices and changes yourself using these technologies, and people used to joke about you know asking Dr. Google. But Dr. Google is not terrible, particularly if you go to WebMD. And, you know, when I look at long-range change, many of the regulations that exist around healthcare delivery were formed at a point before people had access to good quality information at the touch of their fingertips or when educational levels in general were much, much lower. And many regulations existed because of the incumbent power of particular professional sectors. I’ll give you an example from the United Kingdom. So I have had asthma all of my life. That means I’ve been taking my inhaler, Ventolin, and maybe a steroid inhaler for nearly 50 years. That means that I know … actually, I’ve got more experience, and I—in some sense—know more about it than a general practitioner. LEE: Yeah. AZHAR: And until a few years ago, I would have to go to a general practitioner to get this drug that I’ve been taking for five decades, and there they are, age 30 or whatever it is. And a few years ago, the regulations changed. And now pharmacies can … or pharmacists can prescribe those types of drugs under certain conditions directly. LEE: Right. AZHAR: That was not to do with technology. That was to do with incumbent lock-in. So when we look at the medical industry, the healthcare space, there are some parallels with energy, but there are a few little things that the ability that the consumer has to put in some effort to learn about their condition, but also the fact that some of the regulations that exist just exist because certain professions are powerful. LEE: Yeah, one last question while we’re still on economics. There seems to be a conundrum about productivity and efficiency in healthcare delivery because I’ve never encountered a doctor or a nurse that wants to be able to handle even more patients than they’re doing on a daily basis. And so, you know, if productivity means simply, well, your rounds can now handle 16 patients instead of eight patients, that doesn’t seem necessarily to be a desirable thing. So how can we or should we be thinking about efficiency and productivity since obviously costs are, in most of the developed world, are a huge, huge problem? AZHAR: Yes, and when you described doubling the number of patients on the round, I imagined you buying them all roller skates so they could just whizz around [LAUGHTER] the hospital faster and faster than ever before. We can learn from what happened with the introduction of electricity. Electricity emerged at the end of the 19th century, around the same time that cars were emerging as a product, and car makers were very small and very artisanal. And in the early 1900s, some really smart car makers figured out that electricity was going to be important. And they bought into this technology by putting pendant lights in their workshops so they could “visit more patients.” Right? LEE: Yeah, yeah. AZHAR: They could effectively spend more hours working, and that was a productivity enhancement, and it was noticeable. But, of course, electricity fundamentally changed the productivity by orders of magnitude of people who made cars starting with Henry Ford because he was able to reorganize his factories around the electrical delivery of power and to therefore have the moving assembly line, which 10xed the productivity of that system. So when we think about how AI will affect the clinician, the nurse, the doctor, it’s much easier for us to imagine it as the pendant light that just has them working later … LEE: Right. AZHAR: … than it is to imagine a reconceptualization of the relationship between the clinician and the people they care for. And I’m not sure. I don’t think anybody knows what that looks like. But, you know, I do think that there will be a way that this changes, and you can see that scale out factor. And it may be, Peter, that what we end up doing is we end up saying, OK, because we have these brilliant AIs, there’s a lower level of training and cost and expense that’s required for a broader range of conditions that need treating. And that expands the market, right. That expands the market hugely. It’s what has happened in the market for taxis or ride sharing. The introduction of Uber and the GPS system … LEE: Yup. AZHAR: … has meant many more people now earn their living driving people around in their cars. And at least in London, you had to be reasonably highly trained to do that. So I can see a reorganization is possible. Of course, entrenched interests, the economic flow … and there are many entrenched interests, particularly in the US between the health systems and the, you know, professional bodies that might slow things down. But I think a reimagining is possible. And if I may, I’ll give you one example of that, which is, if you go to countries outside of the US where there are many more sick people per doctor, they have incentives to change the way they deliver their healthcare. And well before there was AI of this quality around, there was a few cases of health systems in India—Aravind Eye Care (opens in new tab) was one, and Narayana Hrudayalaya [now known as Narayana Health (opens in new tab)] was another. And in the latter, they were a cardiac care unit where you couldn’t get enough heart surgeons. LEE: Yeah, yep. AZHAR: So specially trained nurses would operate under the supervision of a single surgeon who would supervise many in parallel. So there are ways of increasing the quality of care, reducing the cost, but it does require a systems change. And we can’t expect a single bright algorithm to do it on its own. LEE: Yeah, really, really interesting. So now let’s get into regulation. And let me start with this question. You know, there are several startup companies I’m aware of that are pushing on, I think, a near-term future possibility that a medical AI for consumer might be allowed, say, to prescribe a medication for you, something that would normally require a doctor or a pharmacist, you know, that is certified in some way, licensed to do. Do you think we’ll get to a point where for certain regulated activities, humans are more or less cut out of the loop? AZHAR: Well, humans would have been in the loop because they would have provided the training data, they would have done the oversight, the quality control. But to your question in general, would we delegate an important decision entirely to a tested set of algorithms? I’m sure we will. We already do that. I delegate less important decisions like, What time should I leave for the airport to Waze. I delegate more important decisions to the automated braking in my car. We will do this at certain levels of risk and threshold. If I come back to my example of prescribing Ventolin. It’s really unclear to me that the prescription of Ventolin, this incredibly benign bronchodilator that is only used by people who’ve been through the asthma process, needs to be prescribed by someone who’s gone through 10 years or 12 years of medical training. And why that couldn’t be prescribed by an algorithm or an AI system. LEE: Right. Yep. Yep. AZHAR: So, you know, I absolutely think that that will be the case and could be the case. I can’t really see what the objections are. And the real issue is where do you draw the line of where you say, “Listen, this is too important,” or “The cost is too great,” or “The side effects are too high,” and therefore this is a point at which we want to have some, you know, human taking personal responsibility, having a liability framework in place, having a sense that there is a person with legal agency who signed off on this decision. And that line I suspect will start fairly low, and what we’d expect to see would be that that would rise progressively over time. LEE: What you just said, that scenario of your personal asthma medication, is really interesting because your personal AI might have the benefit of 50 years of your own experience with that medication. So, in a way, there is at least the data potential for, let’s say, the next prescription to be more personalized and more tailored specifically for you. AZHAR: Yes. Well, let’s dig into this because I think this is super interesting, and we can look at how things have changed. So 15 years ago, if I had a bad asthma attack, which I might have once a year, I would have needed to go and see my general physician. In the UK, it’s very difficult to get an appointment. I would have had to see someone privately who didn’t know me at all because I’ve just walked in off the street, and I would explain my situation. It would take me half a day. Productivity lost. I’ve been miserable for a couple of days with severe wheezing. Then a few years ago the system changed, a protocol changed, and now I have a thing called a rescue pack, which includes prednisolone steroids. It includes something else I’ve just forgotten, and an antibiotic in case I get an upper respiratory tract infection, and I have an “algorithm.” It’s called a protocol. It’s printed out. It’s a flowchart I answer various questions, and then I say, “I’m going to prescribe this to myself.” You know, UK doctors don’t prescribe prednisolone, or prednisone as you may call it in the US, at the drop of a hat, right. It’s a powerful steroid. I can self-administer, and I can now get that repeat prescription without seeing a physician a couple of times a year. And the algorithm, the “AI” is, it’s obviously been done in PowerPoint naturally, and it’s a bunch of arrows. [LAUGHS] Surely, surely, an AI system is going to be more sophisticated, more nuanced, and give me more assurance that I’m making the right decision around something like that. LEE: Yeah. Well, at a minimum, the AI should be able to make that PowerPoint the next time. [LAUGHS] AZHAR: Yeah, yeah. Thank god for Clippy. Yes. LEE: So, you know, I think in our book, we had a lot of certainty about most of the things we’ve discussed here, but one chapter where I felt we really sort of ran out of ideas, frankly, was on regulation. And, you know, what we ended up doing for that chapter is … I can’t remember if it was Carey’s or Zak’s idea, but we asked GPT-4 to have a conversation, a debate with itself [LAUGHS], about regulation. And we made some minor commentary on that. And really, I think we took that approach because we just didn’t have much to offer. By the way, in our defense, I don’t think anyone else had any better ideas anyway. AZHAR: Right. LEE: And so now two years later, do we have better ideas about the need for regulation, the frameworks around which those regulations should be developed, and, you know, what should this look like? AZHAR: So regulation is going to be in some cases very helpful because it provides certainty for the clinician that they’re doing the right thing, that they are still insured for what they’re doing, and it provides some degree of confidence for the patient. And we need to make sure that the claims that are made stand up to quite rigorous levels, where ideally there are RCTs [randomized control trials], and there are the classic set of processes you go through. You do also want to be able to experiment, and so the question is: as a regulator, how can you enable conditions for there to be experimentation? And what is experimentation? Experimentation is learning so that every element of the system can learn from this experience. So finding that space where there can be bit of experimentation, I think, becomes very, very important. And a lot of this is about experience, so I think the first digital therapeutics have received FDA approval, which means there are now people within the FDA who understand how you go about running an approvals process for that, and what that ends up looking like—and of course what we’re very good at doing in this sort of modern hyper-connected world—is we can share that expertise, that knowledge, that experience very, very quickly. So you go from one approval a year to a hundred approvals a year to a thousand approvals a year. So we will then actually, I suspect, need to think about what is it to approve digital therapeutics because, unlike big biological molecules, we can generate these digital therapeutics at the rate of knots [very rapidly]. LEE: Yes. AZHAR: Every road in Hayes Valley in San Francisco, right, is churning out new startups who will want to do things like this. So then, I think about, what does it mean to get approved if indeed it gets approved? But we can also go really far with things that don’t require approval. I come back to my sleep tracking ring. So I’ve been wearing this for a few years, and when I go and see my doctor or I have my annual checkup, one of the first things that he asks is how have I been sleeping. And in fact, I even sync my sleep tracking data to their medical record system, so he’s saying … hearing what I’m saying, but he’s actually pulling up the real data going, This patient’s lying to me again. Of course, I’m very truthful with my doctor, as we should all be. [LAUGHTER] LEE: You know, actually, that brings up a point that consumer-facing health AI has to deal with pop science, bad science, you know, weird stuff that you hear on Reddit. And because one of the things that consumers want to know always is, you know, what’s the truth? AZHAR: Right. LEE: What can I rely on? And I think that somehow feels different than an AI that you actually put in the hands of, let’s say, a licensed practitioner. And so the regulatory issues seem very, very different for these two cases somehow. AZHAR: I agree, they’re very different. And I think for a lot of areas, you will want to build AI systems that are first and foremost for the clinician, even if they have patient extensions, that idea that the clinician can still be with a patient during the week. And you’ll do that anyway because you need the data, and you also need a little bit of a liability shield to have like a sensible person who’s been trained around that. And I think that’s going to be a very important pathway for many AI medical crossovers. We’re going to go through the clinician. LEE: Yeah. AZHAR: But I also do recognize what you say about the, kind of, kooky quackery that exists on Reddit. Although on Creatine, Reddit may yet prove to have been right. [LAUGHTER] LEE: Yeah, that’s right. Yes, yeah, absolutely. Yeah. AZHAR: Sometimes it’s right. And I think that it serves a really good role as a field of extreme experimentation. So if you’re somebody who makes a continuous glucose monitor traditionally given to diabetics but now lots of people will wear them—and sports people will wear them—you probably gathered a lot of extreme tail distribution data by reading the Reddit/biohackers … LEE: Yes. AZHAR: … for the last few years, where people were doing things that you would never want them to really do with the CGM [continuous glucose monitor]. And so I think we shouldn’t understate how important that petri dish can be for helping us learn what could happen next. LEE: Oh, I think it’s absolutely going to be essential and a bigger thing in the future. So I think I just want to close here then with one last question. And I always try to be a little bit provocative with this. And so as you look ahead to what doctors and nurses and patients might be doing two years from now, five years from now, 10 years from now, do you have any kind of firm predictions? AZHAR: I’m going to push the boat out, and I’m going to go further out than closer in. LEE: OK. [LAUGHS] AZHAR: As patients, we will have many, many more touch points and interaction with our biomarkers and our health. We’ll be reading how well we feel through an array of things. And some of them we’ll be wearing directly, like sleep trackers and watches. And so we’ll have a better sense of what’s happening in our lives. It’s like the moment you go from paper bank statements that arrive every month to being able to see your account in real time. LEE: Yes. AZHAR: And I suspect we’ll have … we’ll still have interactions with clinicians because societies that get richer see doctors more, societies that get older see doctors more, and we’re going to be doing both of those over the coming 10 years. But there will be a sense, I think, of continuous health engagement, not in an overbearing way, but just in a sense that we know it’s there, we can check in with it, it’s likely to be data that is compiled on our behalf somewhere centrally and delivered through a user experience that reinforces agency rather than anxiety. And we’re learning how to do that slowly. I don’t think the health apps on our phones and devices have yet quite got that right. And that could help us personalize problems before they arise, and again, I use my experience for things that I’ve tracked really, really well. And I know from my data and from how I’m feeling when I’m on the verge of one of those severe asthma attacks that hits me once a year, and I can take a little bit of preemptive measure, so I think that that will become progressively more common and that sense that we will know our baselines. I mean, when you think about being an athlete, which is something I think about, but I could never ever do, [LAUGHTER] but what happens is you start with your detailed baselines, and that’s what your health coach looks at every three or four months. For most of us, we have no idea of our baselines. You we get our blood pressure measured once a year. We will have baselines, and that will help us on an ongoing basis to better understand and be in control of our health. And then if the product designers get it right, it will be done in a way that doesn’t feel invasive, but it’ll be done in a way that feels enabling. We’ll still be engaging with clinicians augmented by AI systems more and more because they will also have gone up the stack. They won’t be spending their time on just “take two Tylenol and have a lie down” type of engagements because that will be dealt with earlier on in the system. And so we will be there in a very, very different set of relationships. And they will feel that they have different ways of looking after our health. LEE: Azeem, it’s so comforting to hear such a wonderfully optimistic picture of the future of healthcare. And I actually agree with everything you’ve said. Let me just thank you again for joining this conversation. I think it’s been really fascinating. And I think somehow the systemic issues, the systemic issues that you tend to just see with such clarity, I think are going to be the most, kind of, profound drivers of change in the future. So thank you so much. AZHAR: Well, thank you, it’s been my pleasure, Peter, thank you. [TRANSITION MUSIC]   I always think of Azeem as a systems thinker. He’s always able to take the experiences of new technologies at an individual level and then project out to what this could mean for whole organizations and whole societies. In our conversation, I felt that Azeem really connected some of what we learned in a previous episode—for example, from Chrissy Farr—on the evolving consumerization of healthcare to the broader workforce and economic impacts that we’ve heard about from Ethan Mollick.   Azeem’s personal story about managing his asthma was also a great example. You know, he imagines a future, as do I, where personal AI might assist and remember decades of personal experience with a condition like asthma and thereby know more than any human being could possibly know in a deeply personalized and effective way, leading to better care. Azeem’s relentless optimism about our AI future was also so heartening to hear. Both of these conversations leave me really optimistic about the future of AI in medicine. At the same time, it is pretty sobering to realize just how much we’ll all need to change in pretty fundamental and maybe even in radical ways. I think a big insight I got from these conversations is how we interact with machines is going to have to be altered not only at the individual level, but at the company level and maybe even at the societal level. Since my conversation with Ethan and Azeem, there have been some pretty important developments that speak directly to this. Just last week at Build (opens in new tab), which is Microsoft’s yearly developer conference, we announced a slew of AI agent technologies. Our CEO, Satya Nadella, in fact, started his keynote by going online in a GitHub developer environment and then assigning a coding task to an AI agent, basically treating that AI as a full-fledged member of a development team. Other agents, for example, a meeting facilitator, a data analyst, a business researcher, travel agent, and more were also shown during the conference. But pertinent to healthcare specifically, what really blew me away was the demonstration of a healthcare orchestrator agent. And the specific thing here was in Stanford’s cancer treatment center, when they are trying to decide on potentially experimental treatments for cancer patients, they convene a meeting of experts. That is typically called a tumor board. And so this AI healthcare orchestrator agent actually participated as a full-fledged member of a tumor board meeting to help bring data together, make sure that the latest medical knowledge was brought to bear, and to assist in the decision-making around a patient’s cancer treatment. It was pretty amazing. [THEME MUSIC] A big thank-you again to Ethan and Azeem for sharing their knowledge and understanding of the dynamics between AI and society more broadly. And to our listeners, thank you for joining us. I’m really excited for the upcoming episodes, including discussions on medical students’ experiences with AI and AI’s influence on the operation of health systems and public health departments. We hope you’ll continue to tune in. Until next time. [MUSIC FADES]

CDPR releases 37 minutes of Cyberpunk 2077 Switch 2 video - so what have we learned? Previewing the latest port vs Xbox Series S and PlayStation 4. Image credit: CD Projekt RED Face-off by Thomas Morgan Senior Staff Writer, Digital Foundry Published on May 26, 2025 Developer CD Projekt RED has uploaded a generous batch of Switch 2 Cyberpunk 2077 footage week - 37 minutes of direct 4K capture to be exact - giving us an early glimpse at the state of its docked 30fps quality mode. Since it releases on 5th June as a Switch 2 launch title, we don't really have too long to wait to see the real thing in action, though given that this footage comes with no "early build" disclaimer or suchlike it appears CDPR is confident in what it's showing in this material - and for good reason. Poring over all the assets, we have plenty to work with for some preliminary comparisons and even frame-rate analysis. In short, the prospects for this Switch 2 rendition are encouraging overall. In terms of content, CDPR is showing all manner of gameplay: driving, combat, major mission set pieces - you name it, it's included. Some clips even reveal, quite openly, the challenges Switch 2 faces in running such a complex open world game - notably for high speed car action. To its credit, frame-rate delivery at 30 frames per second is strong based on this footage overall, with drops into the 20-30fps range mainly being a problem while speeding through Night City's streets. Especially at points where multiple AI cars clog up its roads, it appears drops and traversal hitches are possible, something we're keen to re-test on its release. It's a positive showing overall, though: on-foot exploration around its markets, the bustling parade sequence teeming with NPCs, and even combat during the Phantom Liberty DLC all run at a perfect 30fps here. In performance terms, this showing is perhaps best put in the context of what's currently possible on last-gen consoles, and also Series S. In re-testing the base PS4 version today for example, it's sobering to find that open world roaming there still plays out with hitching, geometry pop-in and drops to 20-30fps - certainly more than is evident in this Switch 2 footage. Going hands-on with the final build ourselves is a must for any final word on this, but early signs point to fewer glaring issues in traversal and battle. Sit back, relax and enjoy another massive episode of DF Direct Weekly.Watch on YouTube 0:00:00 Introduction 0:00:39 News 1: 37 minutes of Cyberpunk 2077 Switch 2 footage released! 0:18:51 News 2: AMD introduces 9060 XT 0:31:43 News 3: AMD teases "FSR Redstone" 0:44:15 News 4: Doom has hidden performance metrics on Xbox 0:53:38 News 5: Mario Kart World originally planned for Switch 1 1:02:49 News 6: Hellblade 2 coming to PS5 1:11:29 Supporter Q1: What do you make of the Nvidia/Gamers Nexus controversy? 1:19:41 Supporter Q2: If Microsoft is working on an Xbox emulator for Windows, does that signal the end for traditional Xbox consoles? 1:28:56 Supporter Q3: Should Nintendo release a non-portable, home-only Switch 2? 1:35:32 Supporter Q4: Could Switch 2 become a dumping ground for last-gen games? 1:40:29 Supporter Q5: What are your hopes and concerns for Switch 2? On the other hand, Xbox Series S' performance level - in its own 30fps quality mode - is a more aspirational target for Switch 2. We described this version as 'what last-gen should have been' in our original review, thanks to it boasting a broadly rock-solid 30fps experience, and it even went on to receive a 60fps mode post-release. A question mark hovers over the viability of Switch 2's own 40fps performance mode though, where we have no recent assets. More to come on this when we get the game ourselves. In terms of comparisons, image quality is a plus point for Switch 2 when compared to the older PS4 release, and even Series S. Much of this boils down to Nvidia's DLSS upscaling technology being available to Switch 2's Tegra 239 processor. CDPR has already confirmed the use of DLSS to hit a 1080p target in docked play in this case (and a 720p target in handheld mode). However, the actual native pixel counts are typically lower than 1080p - with dynamic scaling taking us to 1280x720 at its nadir during the most extreme 20fps drop on record here while driving. More typically though, numbers like 792p, 810p and 864p crop up at less taxing points in the footage, which is a high enough base pixel count for DLSS to (usually) work its magic and reconstruct a 1080p frame. For perspective, Series S' quality mode renders at a 1296p-1440p range using AMD's FSR 2 as its upscaler (as of a late 2022 patch 1.61, following an upgrade from TAA). Meanwhile, base PS4 continues to run at a 720p-900p range using CDPR's own in-house temporal AA solution. In both cases Switch 2 has an advantage in temporal stability, at least. Even though it runs at a lower pixel count than Series S, DLSS more adeptly cleans up the game's visual noise in certain scenarios compared to FSR 2. Shimmer is minimised across the dampened floors of the market area, while during static moments, fences and character detail up-close resolve with added sharpness via Switch 2's upscaler. To see this content please enable targeting cookies. On the downside, for all its benefits, DLSS does not always hide its lower base pixel input. Driving at speed reveals blocking artefacts on Switch 2, while a later Johnny Silverhand dialogue sequence shows similar break-up around two background NPCs playing basketball. There are some limits on show, then, but it's a respectably competitive result next to Series S all things considered. In fact, it's similar to what we found with Street Fighter 6 comparisons between these two consoles, where Switch 2 pushes a sharper, less visibly noisy frame via DLSS - and despite Capcom's fightert running at a lower native res in that case. Focusing on visual quality, it's a surprise to find Switch 2 is on par with both PS4 and Series S in a great many of its core settings. Paired side-by-side with each, there is scarce evidence of any differences in recreated shots: texture quality is a match, SSR is enabled across the floors, and motion blur is engaged too. There is a difference in ambient occlusion (resulting in thicker pockets of object shading on Switch 2) that needs further investigation - and it's clear that Switch 2 also loses the lens flare effect of the Series S release. That aside, the variance in time of day and NPC placement account for a majority of the differences in the open city - whereas in confined interiors that are perfectly matched, the main difference is again DLSS' impact on image quality. It's a positive peak at CDPR's optimisation efforts so far and it appears to be an improvement on the build I played at Nintendo's Switch 2 event in London last month. We're just ten days away from what's undeniably one of the most technically challenging third party games on Switch 2, and it's certainly a big one for coverage plans at Digital Foundry. In fact as I type this, there's an ongoing effort to bank as much Cyberpunk 2077 footage on other platforms for comparison. Roll on June 5th!