As the Industrial Age took the world by storm, city centers became burgeoning hubs of commerce and activity. New offices and apartments were built higher and higher as density increased …read more

For nearly a century, filmmakers have been questioning what happens when technology becomes so advanced and self-aware that the machines we invent turn against the humans who created them. Artificial intelligence is no longer just a science fiction c

When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works. Microsoft 365 security in the spotlight after Washington Post hack Paul Hill Neowin @ziks_99 · Jun 16, 2025 03:36 EDT The Washington Post has come under cyberattack which saw Microsoft email accounts of several journalists get compromised. The attack, which was discovered last Thursday, is believed to have been conducted by a foreign government due to the topics the journalists cover, including national security, economic policy, and China. Following the hack, the passwords on the affected accounts were reset to prevent access. The fact that a Microsoft work email account was potentially hacked strongly suggests The Washington Post utilizes Microsoft 365, which makes us question the security of Microsoft’s widely used enterprise services. Given that Microsoft 365 is very popular, it is a hot target for attackers. Microsoft's enterprise security offerings and challenges As the investigation into the cyberattack is still ongoing, just how attackers gained access to the accounts of the journalists is unknown, however, Microsoft 365 does have multiple layers of protection that ought to keep journalists safe. One of the security tools is Microsoft Defender for Office 365. If the hackers tried to gain access with malicious links, Defender provides protection against any malicious attachments, links, or email-based phishing attempts with the Advanced Threat Protection feature. Defender also helps to protect against malware that could be used to target journalists at The Washington Post. Another security measure in place is Entra ID which helps enterprises defend against identity-based attacks. Some key features of Entra ID include multi-factor authentication which protects accounts even if a password is compromised, and there are granular access policies that help to limit logins from outside certain locations, unknown devices, or limit which apps can be used. While Microsoft does offer plenty of security technologies with M365, hacks can still take place due to misconfiguration, user-error, or through the exploitation of zero-day vulnerabilities. Essentially, it requires efforts from both Microsoft and the customer to maintain security. Lessons for organizations using Microsoft 365 The incident over at The Washington Post serves as a stark reminder that all organizations, not just news organizations, should audit and strengthen their security setups. Some of the most important security measures you can put in place include mandatory multi-factor authentication (MFA) for all users, especially for privileged accounts; strong password rules such as using letters, numbers, and symbols; regular security awareness training; and installing any security updates in a timely manner. Many of the cyberattacks that we learn about from companies like Microsoft involve hackers taking advantage of the human in the equation, such as being tricked into sharing passwords or sharing sensitive information due to trickery on behalf of the hackers. This highlights that employee training is crucial in protecting systems and that Microsoft’s technologies, as advanced as they are, can’t mitigate all attacks 100 percent of the time. Tags Report a problem with article Follow @NeowinFeed

Monitoring and Support EngineerKeyword StudiosPasig City Metro Manila Philippines2 hours agoApplyWe are seeking an experienced Monitoring and Support Engineer to support the technology initiatives of the IT Infrastructure team at Keywords. The Monitoring and Support Engineer will be responsible for follow-the-sun monitoring of IT infrastructure, prompt reaction on all infrastructure incident, primary resolution of infrastructure incidents and support requests.ResponsibilitiesFull scope of tasks including but not limited to:Ensure that all incidents are handled within SLAs.Initial troubleshooting of Infrastructure incidents.Ensure maximum network & service availability through proactive monitoring.Ensure all the incident and alert tickets contain detailed technical information.Initial troubleshooting of Infrastructure incidents, restoration of services and escalation to level 3 experts if necessary.Participate in Problem management processes.Ensure that all incidents and critical alerts are documented and escalated if necessary.Ensure effective communication to customers about incidents and outages.Identify opportunities for process improvement and efficiency enhancements.Participate in documentation creation to reduce BAU support activities by ensuring that the Service Desks have adequate knowledge articles to close support tickets as level 1.Participate in reporting on monitored data and incidents on company infrastructure.Implement best practices and lessons learned from initiatives and projects to optimize future outcomes.RequirementsBachelor's degree in a relevant technical field or equivalent experience.Understanding of IT Infrastructure technologies, standards and trends.Technical background with 3+ years’ experience in IT operations role delivering IT infrastructure support, monitoring and incident management.Technical knowledge of the Microsoft Stack, Windows networking, Active Directory, ExchangeTechnical knowledge of Network, Storage and Server equipment, virtualization and production setupsExceptional communication and presentation skills, with the ability to articulate technical concepts to non-technical audiences.Strong analytical and problem-solving skills.Strong customer service orientation.BenefitsGreat Place to Work certified for 4 consecutive yearsFlexible work arrangementGlobal exposure Create Your Profile — Game companies can contact you with their relevant job openings. Apply

Secretly working multiple full-time remote jobs may sound like a nightmare — but Americans looking to make their financial dreams come true willingly hustle for it.Over the past two years, Business Insider has interviewed more than two dozen "overemployed" workers, many of whom work in tech roles. They tend to work long hours but say the extra earnings are worth it to pay off student debt, save for an early retirement, and afford expensive vacations and weight-loss drugs. Many started working multiple jobs during the pandemic, when remote job openings soared.One example is Sarah, who's on track to earn about $300,000 this year by secretly working two remote IT jobs. Over the last few years, Sarah said the extra income from job juggling has helped her save more than $100,000 in her 401(k)s, pay off $17,000 in credit card debt, and furnish her home.Sarah, who's in her 50s and lives in the Southeast, said working 12-hour days is worth it for the job security. This security came in handy when she was laid off from one of her jobs last year. She's since found a new second gig."I want to ride this out until I retire," Sarah previously told BI. Business Insider verified her identity, but she asked to use a pseudonym, citing fears of professional repercussions. BI spoke to one boss who caught an employee secretly working another job and fired him. Job juggling could breach some employment contracts and be a fireable offense.Overemployed workers like Sarah told BI how they've landed extra roles, juggled the workload, and stayed under the radar. Some said they rely on tactics like blocking off calendars, using separate devices, minimizing meetings, and sticking to flexible roles with low oversight. While job juggling could have professional repercussions or lead to burnout, and some readers have questioned the ethics of this working arrangement, many workers have told BI they don't feel guilty about their job juggling — and that the financial benefits generally outweigh the downsides and risks. In recent years, some have struggled to land new remote gigs, due in part to hiring slowdowns and return-to-office mandates. Most said they plan to continue pursuing overemployment as long as they can.Read the stories ahead to learn how some Americans have managed the workload, risks, and stress of working multiple jobs — and transformed their finances.

One stock recently impacted by a whirlwind of volatility is Block—the fintech powerhouse behind Square, Cash App, Tidal Music, and more. The company’s COO and CFO, Amrita Ahuja, shares how her team is using new AI tools to find opportunity amid disruption and reach customers left behind by traditional financial systems. Ahuja also shares lessons from the video game industry and discusses Gen Z’s surprising approach to money management.   This is an abridged transcript of an interview from Rapid Response, hosted by Robert Safian, former editor-in-chief of Fast Company. From the team behind the Masters of Scale podcast, Rapid Response features candid conversations with today’s top business leaders navigating real-time challenges. Subscribe to Rapid Response wherever you get your podcasts to ensure you never miss an episode. As a leader, when you’re looking at all of this volatility—the tariffs, consumer sentiment’s been unclear, the stock market’s been all over the place. You guys had a huge one-day drop in early May, and it quickly bounced back. How do you make sense of all these external factors? Yeah, our focus is on what we can control. And ultimately, the thing that we are laser-focused on for our business is product velocity. How quickly can we start small with something, launch something for our customers, and then test and iterate and learn so that ultimately, that something that we’ve launched scales into an important product? I’ll give you an example. Cash App Borrow, which is a product where our customers can get access to a line of credit, often $100, $200, that bridges them from paycheck to paycheck. We know so many Americans are living paycheck to paycheck. That’s a product that we launched about three years ago and have now scaled to serve 9 million actives with $15 billion in credit supply to our customers in a span of a couple short years. The more we can be out testing and launching product at a pace, the more we know we are ultimately delivering value to our customers, and the right things will happen from a stock perspective. Block is a financial services provider. You have Square, the point-of-sale system; the digital wallet Cash App, which you mentioned, which competes with Venmo and Robinhood; and a bunch of others. Then you’ve got the buy-now, pay-later leader Afterpay. You chair Square Financial Services, which is Block’s chartered bank. But you’ve said that in the fintech world, Block is only a little bit fin—that comparatively, it’s more tech. Can you explain what you mean by that? What we think is unique about us is our ability as a technology company to completely change innovation in the space, such that we can help solve systemic issues across credit, payments, commerce, and banking. What that means ultimately is we use technologies like AI and machine learning and data science, and we use these technologies in a unique way, in a way that’s different from a traditional bank. We are able to underwrite those who are often frankly forgotten by the traditional financial ecosystems. Our Square Loans product has almost triple the rate of women-owned businesses that we underwrite. Fifty-eight percent of our loans go to women-owned businesses versus 20% for the industry average. For that Cash App Borrow product I was talking about, 70% of those actives, the 9 million actives that we underwrote, fell below 580 as a FICO score. That’s considered a poor FICO score, and yet 97% of repayments are made on time. And this is because we have unique access to data and these technology and tools which can help us uniquely underwrite this often forgotten customer base. Yeah. I mean, credit—sometimes it’s been blamed for financial excesses. But access to credit is also, as you say, an advantage that’s not available to everyone. Do you have a philosophy between those poles—between risk and opportunity? Or is what you’re saying is that the tech you have allows you to avoid that risk? That’s right. Let’s start with how do the current systems work? It works using inferior data, frankly. It’s more limited data. It’s outdated. Sometimes it’s inaccurate. And it ignores things like someone’s cash flows, the stability of your income, your savings rate, how money moves through your accounts, or how you use alternative forms of credit—like buy now, pay later, which we have in our ecosystem through Afterpay. We have a lot of these signals for our 57 million monthly actives on the Cash App side and for the 4 million small businesses on the Square side, and those, frankly, billions of transaction data points that we have on any given day paired with new technologies. And we intend to continue to be on the forefront of AI, machine learning, and data science to be able to empower more people into the economy. The combination of the superior data and the technologies is what we believe ultimately helps expand access. You have a financial background, but not in the financial services industry. Before Block, you were a video game developer at Activision. Are financial businesses and video games similar? Are there things that are similar about them? There are. There actually are some things that are similar, I will say. There are many things that are unique to each industry. Each industry is incredibly complex. You find that when big technology companies try to do gaming. They’ve taken over the world in many different ways, but they can’t always crack the nut on putting out a great game. Similarly, some of the largest technology companies have dabbled in fintech but haven’t been able to go as deep, so they’re both very nuanced and complex industries. I would say another similarity is that design really matters. Industrial design, the design of products, the interface of products, is absolutely mission-critical to a great game, and it’s absolutely mission-critical to the simplicity and accessibility of our products, be it on Square or Cash App. And then maybe the third thing that I would say is that when I was in gaming, at least the business models were rapidly changing from an intermediary distribution mechanism, like releasing a game once and then selling it through a retailer, to an always-on, direct-to-consumer connection. And similarly with banking, people don’t want to bank from 9 to 5, six days a week. They want 24/7 access to their money and the ability to, again, grow their financial livelihood, move their money around seamlessly. So, some similarities are there in that shift to an intermediary model or a slower model to an always-on, direct-to-consumer connection. Part of your target audience or your target customer base at Block are Gen Z folks. Did you learn things at Activision about Gen Z that has been useful? Are there things that businesses misunderstand about younger generations still? What we’ve learned is that Gen Z, millennial customers, aren’t going to do things the way their parents did. Some of our stats show that 63% of Gen Z customers have moved away from traditional credit cards, and over 80% are skeptical of them. Which means they’re not using a credit card to manage expenses; they’re using a debit card, but then layering on on a transaction-by-transaction basis. Or again, using tools like buy now, pay later, or Cash App Borrow, the means in which they’re managing their consistent cash flows. So that’s an example of how things are changing, and you’ve got to get up to speed with how the next generation of customers expects to manage their money.

This article was originally published on GameDevDigest.comEnjoy!What was Radiant AI, anyway? - A ridiculously deep dive into Oblivion's controversial AI system and its legacyblog.paavo.meConsider The Horse Game - No I don’t think every dev should make a horse game (unlike horror, which I still think everyone should at least one). But I do think every developer should at least look at them, maybe even play one because, it is very important that you understand the importance of genre, fandom, and how visibility works. Even if you are not making a horse game, the lessons you can learn by looking at this sub genre are very similar to other genres, just not as blatantly clear as they are with horse games.howtomarketagame.comMaking a killing: The playful 2D terror of Psycasso® - I sat down with lead developer Benjamin Lavender and Omni, designer and producer, to talk about this playfully gory game that gives a classic retro style and a fresh (if gruesome) twist.UnityIntroduction to Asset Manager transfer methods in Unity - Unity's Asset Manager is a user-friendly digital asset management platform supporting over 70 file formats to help teams centralize, organize, discover, and use assets seamlessly across projects. It reduces redundant work by design, making cross-team collaboration smoother and accelerating production workflows.UnityVideosRules of the Game: Five Tricks of Highly Effective Designers - Every working designer has them: unique techniques or "tricks" that they use when crafting gameplay. Sure, there's the general game design wisdom that everyone agrees on and can be found in many a game design book, but experienced game designers often have very specific rules that are personal to them, techniques that not everyone knows about or even agrees with. In this GDC 2015 session, five experienced game designers join the stage for 10 minutes each to share one game design "trick" that they use.Game Developers ConferenceBinding of Isaac Style Room Generator in Unity [Full Tutorial] - Our third part in the series - making the rooms!Game Dev GarnetIntroduction to Unity Behavior | Unity Tutorial - In this video you'll become familiar with the core concepts of Unity Behavior, including a live example.LlamAcademyHow I got my demo ready for Steam Next Fest - It's Steam Next Fest, and I've got a game in the showcase. So here are 7 tips for making the most of this demo sharing festival.Game Maker's ToolkitOptimizing lighting in Projekt Z: Beyond Order - 314 Arts studio lead and founder Justin Miersch discuss how the team used the Screen Space Global Illumination feature in Unity’s High Definition Render Pipeline (HDRP), along with the Unity Profiler and Timeline to overcome the lighting challenges they faced in building Projekt Z: Beyond Order.UnityMemory Arenas in Unity: Heap Allocation Without the GC - In this video, we explore how to build a custom memory arena in Unity using unsafe code and manual heap allocation. You’ll learn how to allocate raw memory for temporary graph-like structures—such as crafting trees or decision planners—without triggering the garbage collector. We’ll walk through the concept of stack frames, translate that to heap-based arena allocation, and implement a fast, disposable system that gives you full control over memory layout and lifetime. Perfect for performance-critical systems where GC spikes aren’t acceptable.git-amendCloth Animation Using The Compute Shader - In this video, we dive into cloth simulation using OpenGL compute shaders. By applying simple mathematical equations, we’ll achieve smooth, dynamic movement. We'll explore particle-based simulation, tackle synchronization challenges with double buffering, and optimize rendering using triangle strips for efficient memory usage. Whether you're familiar with compute shaders or just getting started, this is the perfect way to step up your real-time graphics skills!OGLDEVHow we're designing games for a broader audience - Our games are too hardBiteMe GamesAssetsLearn Game Dev - Unity, Godot, Unreal, Gamemaker, Blender & C# - Make games like a pro.Passionate about video games? Then start making your own! Our latest bundle will help you learn vital game development skills. Master the most popular creation platforms like Unity, Godot, Unreal, GameMaker, Blender, and C#—now that’s a sharp-lookin’ bundle! Build a 2.5D farming RPG with Unreal Engine, create a micro turn-based RPG in Godot, explore game optimization, and so much more.__Big Bang Unreal & Unity Asset Packs Bundle - 5000+ unrivaled assets in one bundle. Calling all game devs—build your worlds with this gigantic bundle of over 5000 assets, including realistic and stylized environments, SFX packs, and powerful tools. Perfect for hobbyists, beginners, and professional developers alike, you'll gain access to essential resources, tutorials, and beta-testing–ready content to start building immediately. The experts at Leartes Studios have curated an amazing library packed with value, featuring environments, VFX packs, and tutorial courses on Unreal Engine, Blender, Substance Painter, and ZBrush. Get the assets you need to bring your game to life—and help support One Tree Planted with your purchase! This bundle provides Unity Asset Store keys directly with your purchase, and FAB keys via redemption through Cosmos, if the product is available on those platforms.Humble Bundle AffiliateGameplay Tools 50% Off - Core systems, half the price. Get pro-grade tools to power your gameplay—combat, cutscenes, UI, and more. Including: HTrace: World Space Global Illumination, VFX Graph - Ultra Mega Pack - Vol.1, Magic Animation Blend, Utility Intelligence (v2): Utility AI Framework for Unity 6, Build for iOS/macOS on Windows>?Unity AffiliateHi guys, I created a website about 6 years in which I host all my field recordings and foley sounds. All free to download and use CC0. There is currently 50+ packs with 1000's of sounds and hours of field recordings all perfect for game SFX and UI. - I think game designers can benefit from a wide range of sounds on the site, especially those that enhance immersion and atmosphere.signaturesounds.orgSmartAddresser - Automate Addressing, Labeling, and Version Control for Unity's Addressable Asset System.CyberAgentGameEntertainment Open SourceEasyCS - EasyCS is an easy-to-use and flexible framework for Unity, adopting a Data-Driven Entity & Actor-Component approach. It bridges Unity's classic OOP with powerful data-oriented patterns, without forcing a complete ECS paradigm shift or a mindset change. Build smarter, not harder.Watcher3056 Open SourceBinding-Of-Isaac_Map-Generator - Binding of Isaac map generator for Unity2DGarnetKane99 Open SourceHelion - A modern fast paced Doom FPS engineHelion-Engine Open SourcePixelationFx - Pixelation post effect for Unity UrpNullTale Open SourceExtreme Add-Ons Bundle For Blender & ZBrush - Extraordinary quality—Extreme add-ons Get quality add-ons for Blender and ZBrush with our latest bundle! We’ve teamed up with the pros at FlippedNormals to deliver a gigantic library of powerful tools for your next game development project. Add new life to your creative work with standout assets like Real-time Hair ZBrush Plugin, Physical Starlight and Atmosphere, Easy Mesh ZBrush Plugin, and more. Get the add-ons you need to bring color and individuality to your next project—and help support Extra Life with your purchase!Humble Bundle AffiliateShop up to 50% off Gabriel Aguiar Prod - Publisher Sale - Gabriel Aguiar Prod. is best known for his extensive VFX assets that help many developers prototype and ship games with special effects. His support and educational material are also invaluable resources for the game dev community. PLUS get VFX Graph - Stylized Fire - Vol. 1 for FREE with code GAP2025Unity AffiliateSpotlightDream Garden - Dream Garden is a simulation game about building tiny cute garden dioramas. A large selection of tools, plants, decorations and customization awaits you. Try all of them and create your dream garden.[You can find it on Steam]Campfire StudioMy game, Call Of Dookie. Demo available on SteamYou can subscribe to the free weekly newsletter on GameDevDigest.comThis post includes affiliate links; I may receive compensation if you purchase products or services from the different links provided in this article.

Deal  When you purchase through links on our site, we may earn an affiliate commission. Here’s how it works. The 2025 Complete Splunk Beginner Bundle is now 25% off Neowin Deals · Jun 14, 2025 16:00 EDT Today's highlighted deal comes via Neowin Deals store, where you can save 75% on The 2025 Complete Splunk Beginner Bundle Splunk is a powerful data platform used to gather information from multiple sources and index it for efficient access. You can then use collected data to create visualizations, analytics, and a variety of automated and security-related functions. With its web-style interface, Splunk is easy to use and is utilized by many companies worldwide. What's Included: Splunk Fundamentals for Effective Management of SOC and SIEM Oak Academy 38 Lessons (3.5h) Lifetime $20.00 Value Splunk | Splunk Core Certified User Certification Prep Lab Oak Academy, 63 Lessons (6h),Lifetime, $20.00 Value Splunk | Splunk Core Certified Power User SPLK 1002 Prep Oak Academy, 53 Lessons (5.5h), Lifetime, $20.00 Value Splunk| Splunk Enterprise Certified Admin Certification Prep Oak Academy, 68 Lessons (8.5h), Lifetime, $20.00 Value Requirements Basic understanding of IT and networking concepts Familiarity with Linux and Windows operating systems A computer with internet access for hands-on practice Good to Know Length of time users can access this course: lifetime Access options: desktop or mobile Redemption deadline: redeem your code within 30 days of purchase Experience level required: all levels Certificate of Completion ONLY Updates included Closed captioning NOT available NOT downloadable for offline viewing Learn more about our Lifetime deals here! Lifetime access to this 2025 Complete Splunk Beginner Bundle normally costs $80, but this deal can be yours for just $19.99, that's a saving of $60. For full terms, specifications, and info, click the link below. Although priced in U.S. dollars, this deal is available for digital purchase worldwide. We post these because we earn commission on each sale so as not to rely solely on advertising, which many of our readers block. It all helps toward paying staff reporters, servers and hosting costs. Other ways to support Neowin Whitelist Neowin by not blocking our ads Create a free member account to see fewer ads Make a donation to support our day to day running costs Subscribe to Neowin - for $14 a year, or $28 a year for an ad-free experience Disclosure: Neowin benefits from revenue of each sale made through our branded deals site powered by StackCommerce. Tags Report a problem with article Follow @NeowinFeed

Reading Time: 9 minutes In the ever-evolving landscape of customer engagement, staying ahead of the curve is not just advantageous, it’s essential. That’s why, for Chapter 7 of “The Customer Engagement Book: Adapt or Die,” we sat down with Mirela Cialai, a seasoned expert in CRM and Martech strategies at brands like Equinox. Mirela brings a wealth of knowledge in aligning technology roadmaps with business goals, shifting organizational focuses from acquisition to retention, and leveraging hyper-personalization to drive success. In this interview, Mirela dives deep into building robust customer engagement technology roadmaps. She unveils the “PAPER” framework—Plan, Audit, Prioritize, Execute, Refine—a simple yet effective strategy for marketers. You’ll gain insights into identifying gaps in your Martech stack, ensuring data accuracy, and prioritizing initiatives that deliver the greatest impact and ROI. Whether you’re navigating data silos, striving for cross-functional alignment, or aiming for seamless tech integration, Mirela’s expertise provides practical solutions and actionable takeaways.   Mirela Cialai Q&A Interview 1. How do you define the vision for a customer engagement platform roadmap in alignment with the broader business goals? Can you share any examples of successful visions from your experience? Defining the vision for the roadmap in alignment with the broader business goals involves creating a strategic framework that connects the team’s objectives with the organization’s overarching mission or primary objectives. This could be revenue growth, customer retention, market expansion, or operational efficiency. We then break down these goals into actionable areas where the team can contribute, such as improving engagement, increasing lifetime value, or driving acquisition. We articulate how the team will support business goals by defining the KPIs that link CRM outcomes — the team’s outcomes — to business goals. In a previous role, the CRM team I was leading faced significant challenges due to the lack of attribution capabilities and a reliance on surface-level metrics such as open rates and click-through rates to measure performance. This approach made it difficult to quantify the impact of our efforts on broader business objectives such as revenue growth. Recognizing this gap, I worked on defining a vision for the CRM team to address these shortcomings. Our vision was to drive measurable growth through enhanced data accuracy and improved attribution capabilities, which allowed us to deliver targeted, data-driven, and personalized customer experiences. To bring this vision to life, I developed a roadmap that focused on first improving data accuracy, building our attribution capabilities, and delivering personalization at scale. By aligning the vision with these strategic priorities, we were able to demonstrate the tangible impact of our efforts on the key business goals. 2. What steps did you take to ensure data accuracy? The data team was very diligent in ensuring that our data warehouse had accurate data. So taking that as the source of truth, we started cleaning the data in all the other platforms that were integrated with our data warehouse — our CRM platform, our attribution analytics platform, etc. That’s where we started, looking at all the different integrations and ensuring that the data flows were correct and that we had all the right flows in place. And also validating and cleaning our email database — that helped, having more accurate data. 3. How do you recommend shifting organizational focus from acquisition to retention within a customer engagement strategy? Shifting an organization’s focus from acquisition to retention requires a cultural and strategic shift, emphasizing the immense value that existing customers bring to long-term growth and profitability. I would start by quantifying the value of retention, showcasing how retaining customers is significantly more cost-effective than acquiring new ones. Research consistently shows that increasing retention rates by just 5% can boost profits by at least 25 to 95%. This data helps make a compelling case to stakeholders about the importance of prioritizing retention. Next, I would link retention to core business goals by demonstrating how enhancing customer lifetime value and loyalty can directly drive revenue growth. This involves shifting the organization’s focus to retention-specific metrics such as churn rate, repeat purchase rate, and customer LTV. These metrics provide actionable insights into customer behaviors and highlight the financial impact of retention initiatives, ensuring alignment with the broader company objectives. By framing retention as a driver of sustainable growth, the organization can see it not as a competing priority, but as a complementary strategy to acquisition, ultimately leading to a more balanced and effective customer engagement strategy. 4. What are the key steps in analyzing a brand’s current Martech stack capabilities to identify gaps and opportunities for improvement? Developing a clear understanding of the Martech stack’s current state and ensuring it aligns with a brand’s strategic needs and future goals requires a structured and strategic approach. The process begins with defining what success looks like in terms of technology capabilities such as scalability, integration, automation, and data accessibility, and linking these capabilities directly to the brand’s broader business objectives. I start by doing an inventory of all tools currently in use, including their purpose, owner, and key functionalities, assessing if these tools are being used to their full potential or if there are features that remain unused, and reviewing how well tools integrate with one another and with our core systems, the data warehouse. Also, comparing the capabilities of each tool and results against industry standards and competitor practices and looking for missing functionalities such as personalization, omnichannel orchestration, or advanced analytics, and identifying overlapping tools that could be consolidated to save costs and streamline workflows. Finally, review the costs of the current tools against their impact on business outcomes and identify technologies that could reduce costs, increase efficiency, or deliver higher ROI through enhanced capabilities. Establish a regular review cycle for the Martech stack to ensure it evolves alongside the business and the technological landscape. 5. How do you evaluate whether a company’s tech stack can support innovative customer-focused campaigns, and what red flags should marketers look out for? I recommend taking a structured approach and first ensure there is seamless integration across all tools to support a unified customer view and data sharing across the different channels. Determine if the stack can handle increasing data volumes, larger audiences, and additional channels as the campaigns grow, and check if it supports dynamic content, behavior-based triggers, and advanced segmentation and can process and act on data in real time through emerging technologies like AI/ML predictive analytics to enable marketers to launch responsive and timely campaigns. Most importantly, we need to ensure that the stack offers robust reporting tools that provide actionable insights, allowing teams to track performance and optimize campaigns. Some of the red flags are: data silos where customer data is fragmented across platforms and not easily accessible or integrated, inability to process or respond to customer behavior in real time, a reliance on manual intervention for tasks like segmentation, data extraction, campaign deployment, and poor scalability. If the stack struggles with growing data volumes or expanding to new channels, it won’t support the company’s evolving needs. 6. What role do hyper-personalization and timely communication play in a successful customer engagement strategy? How do you ensure they’re built into the technology roadmap? Hyper-personalization and timely communication are essential components of a successful customer engagement strategy because they create meaningful, relevant, and impactful experiences that deepen the relationship with customers, enhance loyalty, and drive business outcomes. Hyper-personalization leverages data to deliver tailored content that resonates with each individual based on their preferences, behavior, or past interactions, and timely communication ensures these personalized interactions occur at the most relevant moments, which ultimately increases their impact. Customers are more likely to engage with messages that feel relevant and align with their needs, and real-time triggers such as cart abandonment or post-purchase upsells capitalize on moments when customers are most likely to convert. By embedding these capabilities into the roadmap through data integration, AI-driven insights, automation, and continuous optimization, we can deliver impactful, relevant, and timely experiences that foster deeper customer relationships and drive long-term success. 7. What’s your approach to breaking down the customer engagement technology roadmap into manageable phases? How do you prioritize the initiatives? To create a manageable roadmap, we need to divide it into distinct phases, starting with building the foundation by addressing data cleanup, system integrations, and establishing metrics, which lays the groundwork for success. Next, we can focus on early wins and quick impact by launching behavior-based campaigns, automating workflows, and improving personalization to drive immediate value. Then we can move to optimization and expansion, incorporating predictive analytics, cross-channel orchestration, and refined attribution models to enhance our capabilities. Finally, prioritize innovation and scalability, leveraging AI/ML for hyper-personalization, scaling campaigns to new markets, and ensuring the system is equipped for future growth. By starting with foundational projects, delivering quick wins, and building towards scalable innovation, we can drive measurable outcomes while maintaining our agility to adapt to evolving needs. In terms of prioritizing initiatives effectively, I would focus on projects that deliver the greatest impact on business goals, on customer experience and ROI, while we consider feasibility, urgency, and resource availability. In the past, I’ve used frameworks like Impact Effort Matrix to identify the high-impact, low-effort initiatives and ensure that the most critical projects are addressed first. 8. How do you ensure cross-functional alignment around this roadmap? What processes have worked best for you? Ensuring cross-functional alignment requires clear communication, collaborative planning, and shared accountability. We need to establish a shared understanding of the roadmap’s purpose and how it ties to the company’s overall goals by clearly articulating the “why” behind the roadmap and how each team can contribute to its success. To foster buy-in and ensure the roadmap reflects diverse perspectives and needs, we need to involve all stakeholders early on during the roadmap development and clearly outline each team’s role in executing the roadmap to ensure accountability across the different teams. To keep teams informed and aligned, we use meetings such as roadmap kickoff sessions and regular check-ins to share updates, address challenges collaboratively, and celebrate milestones together. 9. If you were to outline a simple framework for marketers to follow when building a customer engagement technology roadmap, what would it look like? A simple framework for marketers to follow when building the roadmap can be summarized in five clear steps: Plan, Audit, Prioritize, Execute, and Refine. In one word: PAPER. Here’s how it breaks down. Plan: We lay the groundwork for the roadmap by defining the CRM strategy and aligning it with the business goals. Audit: We evaluate the current state of our CRM capabilities. We conduct a comprehensive assessment of our tools, our data, the processes, and team workflows to identify any potential gaps. Prioritize: initiatives based on impact, feasibility, and ROI potential. Execute: by implementing the roadmap in manageable phases. Refine: by continuously improving CRM performance and refining the roadmap. So the PAPER framework — Plan, Audit, Prioritize, Execute, and Refine — provides a structured, iterative approach allowing marketers to create a scalable and impactful customer engagement strategy. 10. What are the most common challenges marketers face in creating or executing a customer engagement strategy, and how can they address these effectively? The most critical is when the customer data is siloed across different tools and platforms, making it very difficult to get a unified view of the customer. This limits the ability to deliver personalized and consistent experiences. The solution is to invest in tools that can centralize data from all touchpoints and ensure seamless integration between different platforms to create a single source of truth. Another challenge is the lack of clear metrics and ROI measurement and the inability to connect engagement efforts to tangible business outcomes, making it very hard to justify investment or optimize strategies. The solution for that is to define clear KPIs at the outset and use attribution models to link customer interactions to revenue and other key outcomes. Overcoming internal silos is another challenge where there is misalignment between teams, which can lead to inconsistent messaging and delayed execution. A solution to this is to foster cross-functional collaboration through shared goals, regular communication, and joint planning sessions. Besides these, other challenges marketers can face are delivering personalization at scale, keeping up with changing customer expectations, resource and budget constraints, resistance to change, and others. While creating and executing a customer engagement strategy can be challenging, these obstacles can be addressed through strategic planning, leveraging the right tools, fostering collaboration, and staying adaptable to customer needs and industry trends. By tackling these challenges proactively, marketers can deliver impactful customer-centric strategies that drive long-term success. 11. What are the top takeaways or lessons that you’ve learned from building customer engagement technology roadmaps that others should keep in mind? I would say one of the most important takeaways is to ensure that the roadmap directly supports the company’s broader objectives. Whether the focus is on retention, customer lifetime value, or revenue growth, the roadmap must bridge the gap between high-level business goals and actionable initiatives. Another important lesson: The roadmap is only as effective as the data and systems it’s built upon. I’ve learned the importance of prioritizing foundational elements like data cleanup, integrations, and governance before tackling advanced initiatives like personalization or predictive analytics. Skipping this step can lead to inefficiencies or missed opportunities later on. A Customer Engagement Roadmap is a strategic tool that evolves alongside the business and its customers. So by aligning with business goals, building a solid foundation, focusing on impact, fostering collaboration, and remaining adaptable, you can create a roadmap that delivers measurable results and meaningful customer experiences.     This interview Q&A was hosted with Mirela Cialai, Director of CRM & MarTech at Equinox, for Chapter 7 of The Customer Engagement Book: Adapt or Die. Download the PDF or request a physical copy of the book here. The post Mirela Cialai Q&A: Customer Engagement Book Interview appeared first on MoEngage.

Transcript [MUSIC]      [BOOK PASSAGE]   PETER LEE: “In ‘The Little Black Bag,’ a classic science fiction story, a high-tech doctor’s kit of the future is accidentally transported back to the 1950s, into the shaky hands of a washed-up, alcoholic doctor. The ultimate medical tool, it redeems the doctor wielding it, allowing him to practice gratifyingly heroic medicine. … The tale ends badly for the doctor and his treacherous assistant, but it offered a picture of how advanced technology could transform medicine—powerful when it was written nearly 75 years ago and still so today. What would be the Al equivalent of that little black bag? At this moment when new capabilities are emerging, how do we imagine them into medicine?”   [END OF BOOK PASSAGE]     [THEME MUSIC]     This is The AI Revolution in Medicine, Revisited. I’m your host, Peter Lee.    Shortly after OpenAI’s GPT-4 was publicly released, Carey Goldberg, Dr. Zak Kohane, and I published The AI Revolution in Medicine to help educate the world of healthcare and medical research about the transformative impact this new generative AI technology could have. But because we wrote the book when GPT-4 was still a secret, we had to speculate. Now, two years later, what did we get right, and what did we get wrong?     In this series, we’ll talk to clinicians, patients, hospital administrators, and others to understand the reality of AI in the field and where we go from here.   [THEME MUSIC FADES] The book passage I read at the top is from “Chapter 10: The Big Black Bag.”  In imagining AI in medicine, Carey, Zak, and I included in our book two fictional accounts. In the first, a medical resident consults GPT-4 on her personal phone as the patient in front of her crashes. Within seconds, it offers an alternate response based on recent literature. In the second account, a 90-year-old woman with several chronic conditions is living independently and receiving near-constant medical support from an AI aide.    In our conversations with the guests we’ve spoken to so far, we’ve caught a glimpse of these predicted futures, seeing how clinicians and patients are actually using AI today and how developers are leveraging the technology in the healthcare products and services they’re creating. In fact, that first fictional account isn’t so fictional after all, as most of the doctors in the real world actually appear to be using AI at least occasionally—and sometimes much more than occasionally—to help in their daily clinical work. And as for the second fictional account, which is more of a science fiction account, it seems we are indeed on the verge of a new way of delivering and receiving healthcare, though the future is still very much open.  As we continue to examine the current state of AI in healthcare and its potential to transform the field, I’m pleased to welcome Bill Gates and Sébastien Bubeck.   Bill may be best known as the co-founder of Microsoft, having created the company with his childhood friend Paul Allen in 1975. He’s now the founder of Breakthrough Energy, which aims to advance clean energy innovation, and TerraPower, a company developing groundbreaking nuclear energy and science technologies. He also chairs the world’s largest philanthropic organization, the Gates Foundation, and focuses on solving a variety of health challenges around the globe and here at home.  Sébastien is a research lead at OpenAI. He was previously a distinguished scientist, vice president of AI, and a colleague of mine here at Microsoft, where his work included spearheading the development of the family of small language models known as Phi. While at Microsoft, he also coauthored the discussion-provoking 2023 paper “Sparks of Artificial General Intelligence,” which presented the results of early experiments with GPT-4 conducted by a small team from Microsoft Research.    [TRANSITION MUSIC]   Here’s my conversation with Bill Gates and Sébastien Bubeck.  LEE: Bill, welcome.  BILL GATES: Thank you.  LEE: Seb …  SÉBASTIEN BUBECK: Yeah. Hi, hi, Peter. Nice to be here.  LEE: You know, one of the things that I’ve been doing just to get the conversation warmed up is to talk about origin stories, and what I mean about origin stories is, you know, what was the first contact that you had with large language models or the concept of generative AI that convinced you or made you think that something really important was happening?  And so, Bill, I think I’ve heard the story about, you know, the time when the OpenAI folks—Sam Altman, Greg Brockman, and others—showed you something, but could we hear from you what those early encounters were like and what was going through your mind?   GATES: Well, I’d been visiting OpenAI soon after it was created to see things like GPT-2 and to see the little arm they had that was trying to match human manipulation and, you know, looking at their games like Dota that they were trying to get as good as human play. And honestly, I didn’t think the language model stuff they were doing, even when they got to GPT-3, would show the ability to learn, you know, in the same sense that a human reads a biology book and is able to take that knowledge and access it not only to pass a test but also to create new medicines.  And so my challenge to them was that if their LLM could get a five on the advanced placement biology test, then I would say, OK, it took biologic knowledge and encoded it in an accessible way and that I didn’t expect them to do that very quickly but it would be profound.   And it was only about six months after I challenged them to do that, that an early version of GPT-4 they brought up to a dinner at my house, and in fact, it answered most of the questions that night very well. The one it got totally wrong, we were … because it was so good, we kept thinking, Oh, we must be wrong. It turned out it was a math weakness [LAUGHTER] that, you know, we later understood that that was an area of, weirdly, of incredible weakness of those early models. But, you know, that was when I realized, OK, the age of cheap intelligence was at its beginning.  LEE: Yeah. So I guess it seems like you had something similar to me in that my first encounters, I actually harbored some skepticism. Is it fair to say you were skeptical before that?  GATES: Well, the idea that we’ve figured out how to encode and access knowledge in this very deep sense without even understanding the nature of the encoding, …  LEE: Right.   GATES: … that is a bit weird.   LEE: Yeah.  GATES: We have an algorithm that creates the computation, but even say, OK, where is the president’s birthday stored in there? Where is this fact stored in there? The fact that even now when we’re playing around, getting a little bit more sense of it, it’s opaque to us what the semantic encoding is, it’s, kind of, amazing to me. I thought the invention of knowledge storage would be an explicit way of encoding knowledge, not an implicit statistical training.  LEE: Yeah, yeah. All right. So, Seb, you know, on this same topic, you know, I got—as we say at Microsoft—I got pulled into the tent. [LAUGHS]  BUBECK: Yes.   LEE: Because this was a very secret project. And then, um, I had the opportunity to select a small number of researchers in MSR [Microsoft Research] to join and start investigating this thing seriously. And the first person I pulled in was you.  BUBECK: Yeah.  LEE: And so what were your first encounters? Because I actually don’t remember what happened then.  BUBECK: Oh, I remember it very well. [LAUGHS] My first encounter with GPT-4 was in a meeting with the two of you, actually. But my kind of first contact, the first moment where I realized that something was happening with generative AI, was before that. And I agree with Bill that I also wasn’t too impressed by GPT-3.  I though that it was kind of, you know, very naturally mimicking the web, sort of parroting what was written there in a nice way. Still in a way which seemed very impressive. But it wasn’t really intelligent in any way. But shortly after GPT-3, there was a model before GPT-4 that really shocked me, and this was the first image generation model, DALL-E 1.  So that was in 2021. And I will forever remember the press release of OpenAI where they had this prompt of an avocado chair and then you had this image of the avocado chair. [LAUGHTER] And what really shocked me is that clearly the model kind of “understood” what is a chair, what is an avocado, and was able to merge those concepts.  So this was really, to me, the first moment where I saw some understanding in those models.   LEE: So this was, just to get the timing right, that was before I pulled you into the tent.  BUBECK: That was before. That was like a year before.  LEE: Right.   BUBECK: And now I will tell you how, you know, we went from that moment to the meeting with the two of you and GPT-4.  So once I saw this kind of understanding, I thought, OK, fine. It understands concept, but it’s still not able to reason. It cannot—as, you know, Bill was saying—it cannot learn from your document. It cannot reason.   So I set out to try to prove that. You know, this is what I was in the business of at the time, trying to prove things in mathematics. So I was trying to prove that basically autoregressive transformers could never reason. So I was trying to prove this. And after a year of work, I had something reasonable to show. And so I had the meeting with the two of you, and I had this example where I wanted to say, there is no way that an LLM is going to be able to do x.  And then as soon as I … I don’t know if you remember, Bill. But as soon as I said that, you said, oh, but wait a second. I had, you know, the OpenAI crew at my house recently, and they showed me a new model. Why don’t we ask this new model this question?   LEE: Yeah. BUBECK: And we did, and it solved it on the spot. And that really, honestly, just changed my life. Like, you know, I had been working for a year trying to say that this was impossible. And just right there, it was shown to be possible.   LEE: [LAUGHS] One of the very first things I got interested in—because I was really thinking a lot about healthcare—was healthcare and medicine.  And I don’t know if the two of you remember, but I ended up doing a lot of tests. I ran through, you know, step one and step two of the US Medical Licensing Exam. Did a whole bunch of other things. I wrote this big report. It was, you know, I can’t remember … a couple hundred pages.   And I needed to share this with someone. I didn’t … there weren’t too many people I could share it with. So I sent, I think, a copy to you, Bill. Sent a copy to you, Seb.   I hardly slept for about a week putting that report together. And, yeah, and I kept working on it. But I was far from alone. I think everyone who was in the tent, so to speak, in those early days was going through something pretty similar. All right. So I think … of course, a lot of what I put in the report also ended up being examples that made it into the book.  But the main purpose of this conversation isn’t to reminisce about [LAUGHS] or indulge in those reminiscences but to talk about what’s happening in healthcare and medicine. And, you know, as I said, we wrote this book. We did it very, very quickly. Seb, you helped. Bill, you know, you provided a review and some endorsements.  But, you know, honestly, we didn’t know what we were talking about because no one had access to this thing. And so we just made a bunch of guesses. So really, the whole thing I wanted to probe with the two of you is, now with two years of experience out in the world, what, you know, what do we think is happening today?  You know, is AI actually having an impact, positive or negative, on healthcare and medicine? And what do we now think is going to happen in the next two years, five years, or 10 years? And so I realize it’s a little bit too abstract to just ask it that way. So let me just try to narrow the discussion and guide us a little bit.   Um, the kind of administrative and clerical work, paperwork, around healthcare—and we made a lot of guesses about that—that appears to be going well, but, you know, Bill, I know we’ve discussed that sometimes that you think there ought to be a lot more going on. Do you have a viewpoint on how AI is actually finding its way into reducing paperwork?  GATES: Well, I’m stunned … I don’t think there should be a patient-doctor meeting where the AI is not sitting in and both transcribing, offering to help with the paperwork, and even making suggestions, although the doctor will be the one, you know, who makes the final decision about the diagnosis and whatever prescription gets done.   It’s so helpful. You know, when that patient goes home and their, you know, son who wants to understand what happened has some questions, that AI should be available to continue that conversation. And the way you can improve that experience and streamline things and, you know, involve the people who advise you. I don’t understand why that’s not more adopted, because there you still have the human in the loop making that final decision.  But even for, like, follow-up calls to make sure the patient did things, to understand if they have concerns and knowing when to escalate back to the doctor, the benefit is incredible. And, you know, that thing is ready for prime time. That paradigm is ready for prime time, in my view.  LEE: Yeah, there are some good products, but it seems like the number one use right now—and we kind of got this from some of the previous guests in previous episodes—is the use of AI just to respond to emails from patients. [LAUGHTER] Does that make sense to you?  BUBECK: Yeah. So maybe I want to second what Bill was saying but maybe take a step back first. You know, two years ago, like, the concept of clinical scribes, which is one of the things that we’re talking about right now, it would have sounded, in fact, it sounded two years ago, borderline dangerous. Because everybody was worried about hallucinations. What happened if you have this AI listening in and then it transcribes, you know, something wrong?  Now, two years later, I think it’s mostly working. And in fact, it is not yet, you know, fully adopted. You’re right. But it is in production. It is used, you know, in many, many places. So this rate of progress is astounding because it wasn’t obvious that we would be able to overcome those obstacles of hallucination. It’s not to say that hallucinations are fully solved. In the case of the closed system, they are.   Now, I think more generally what’s going on in the background is that there is something that we, that certainly I, underestimated, which is this management overhead. So I think the reason why this is not adopted everywhere is really a training and teaching aspect. People need to be taught, like, those systems, how to interact with them.  And one example that I really like, a study that recently appeared where they tried to use ChatGPT for diagnosis and they were comparing doctors without and with ChatGPT (opens in new tab). And the amazing thing … so this was a set of cases where the accuracy of the doctors alone was around 75%. ChatGPT alone was 90%. So that’s already kind of mind blowing. But then the kicker is that doctors with ChatGPT was 80%.   Intelligence alone is not enough. It’s also how it’s presented, how you interact with it. And ChatGPT, it’s an amazing tool. Obviously, I absolutely love it. But it’s not … you don’t want a doctor to have to type in, you know, prompts and use it that way.  It should be, as Bill was saying, kind of running continuously in the background, sending you notifications. And you have to be really careful of the rate at which those notifications are being sent. Because if they are too frequent, then the doctor will learn to ignore them. So you have to … all of those things matter, in fact, at least as much as the level of intelligence of the machine.  LEE: One of the things I think about, Bill, in that scenario that you described, doctors do some thinking about the patient when they write the note. So, you know, I’m always a little uncertain whether it’s actually … you know, you wouldn’t necessarily want to fully automate this, I don’t think. Or at least there needs to be some prompt to the doctor to make sure that the doctor puts some thought into what happened in the encounter with the patient. Does that make sense to you at all?  GATES: At this stage, you know, I’d still put the onus on the doctor to write the conclusions and the summary and not delegate that.  The tradeoffs you make a little bit are somewhat dependent on the situation you’re in. If you’re in Africa, So, yes, the doctor’s still going to have to do a lot of work, but just the quality of letting the patient and the people around them interact and ask questions and have things explained, that alone is such a quality improvement. It’s mind blowing.   LEE: So since you mentioned, you know, Africa—and, of course, this touches on the mission and some of the priorities of the Gates Foundation and this idea of democratization of access to expert medical care—what’s the most interesting stuff going on right now? Are there people and organizations or technologies that are impressing you or that you’re tracking?  GATES: Yeah. So the Gates Foundation has given out a lot of grants to people in Africa doing education, agriculture but more healthcare examples than anything. And the way these things start off, they often start out either being patient-centric in a narrow situation, like, OK, I’m a pregnant woman; talk to me. Or, I have infectious disease symptoms; talk to me. Or they’re connected to a health worker where they’re helping that worker get their job done. And we have lots of pilots out, you know, in both of those cases.   The dream would be eventually to have the thing the patient consults be so broad that it’s like having a doctor available who understands the local things.   LEE: Right.   GATES: We’re not there yet. But over the next two or three years, you know, particularly given the worsening financial constraints against African health systems, where the withdrawal of money has been dramatic, you know, figuring out how to take this—what I sometimes call “free intelligence”—and build a quality health system around that, we will have to be more radical in low-income countries than any rich country is ever going to be.   LEE: Also, there’s maybe a different regulatory environment, so some of those things maybe are easier? Because right now, I think the world hasn’t figured out how to and whether to regulate, let’s say, an AI that might give a medical diagnosis or write a prescription for a medication.  BUBECK: Yeah. I think one issue with this, and it’s also slowing down the deployment of AI in healthcare more generally, is a lack of proper benchmark. Because, you know, you were mentioning the USMLE [United States Medical Licensing Examination], for example. That’s a great test to test human beings and their knowledge of healthcare and medicine. But it’s not a great test to give to an AI.  It’s not asking the right questions. So finding what are the right questions to test whether an AI system is ready to give diagnosis in a constrained setting, that’s a very, very important direction, which to my surprise, is not yet accelerating at the rate that I was hoping for.  LEE: OK, so that gives me an excuse to get more now into the core AI tech because something I’ve discussed with both of you is this issue of what are the right tests. And you both know the very first test I give to any new spin of an LLM is I present a patient, the results—a mythical patient—the results of my physical exam, my mythical physical exam. Maybe some results of some initial labs. And then I present or propose a differential diagnosis. And if you’re not in medicine, a differential diagnosis you can just think of as a prioritized list of the possible diagnoses that fit with all that data. And in that proposed differential, I always intentionally make two mistakes.  I make a textbook technical error in one of the possible elements of the differential diagnosis, and I have an error of omission. And, you know, I just want to know, does the LLM understand what I’m talking about? And all the good ones out there do now. But then I want to know, can it spot the errors? And then most importantly, is it willing to tell me I’m wrong, that I’ve made a mistake?   That last piece seems really hard for AI today. And so let me ask you first, Seb, because at the time of this taping, of course, there was a new spin of GPT-4o last week that became overly sycophantic. In other words, it was actually prone in that test of mine not only to not tell me I’m wrong, but it actually praised me for the creativity of my differential. [LAUGHTER] What’s up with that?  BUBECK: Yeah, I guess it’s a testament to the fact that training those models is still more of an art than a science. So it’s a difficult job. Just to be clear with the audience, we have rolled back that [LAUGHS] version of GPT-4o, so now we don’t have the sycophant version out there.  Yeah, no, it’s a really difficult question. It has to do … as you said, it’s very technical. It has to do with the post-training and how, like, where do you nudge the model? So, you know, there is this very classical by now technique called RLHF [reinforcement learning from human feedback], where you push the model in the direction of a certain reward model. So the reward model is just telling the model, you know, what behavior is good, what behavior is bad.  But this reward model is itself an LLM, and, you know, Bill was saying at the very beginning of the conversation that we don’t really understand how those LLMs deal with concepts like, you know, where is the capital of France located? Things like that. It is the same thing for this reward model. We don’t know why it says that it prefers one output to another, and whether this is correlated with some sycophancy is, you know, something that we discovered basically just now. That if you push too hard in optimization on this reward model, you will get a sycophant model.  So it’s kind of … what I’m trying to say is we became too good at what we were doing, and we ended up, in fact, in a trap of the reward model.  LEE: I mean, you do want … it’s a difficult balance because you do want models to follow your desires and …  BUBECK: It’s a very difficult, very difficult balance.  LEE: So this brings up then the following question for me, which is the extent to which we think we’ll need to have specially trained models for things. So let me start with you, Bill. Do you have a point of view on whether we will need to, you know, quote-unquote take AI models to med school? Have them specially trained? Like, if you were going to deploy something to give medical care in underserved parts of the world, do we need to do something special to create those models?  GATES: We certainly need to teach them the African languages and the unique dialects so that the multimedia interactions are very high quality. We certainly need to teach them the disease prevalence and unique disease patterns like, you know, neglected tropical diseases and malaria. So we need to gather a set of facts that somebody trying to go for a US customer base, you know, wouldn’t necessarily have that in there.  Those two things are actually very straightforward because the additional training time is small. I’d say for the next few years, we’ll also need to do reinforcement learning about the context of being a doctor and how important certain behaviors are. Humans learn over the course of their life to some degree that, I’m in a different context and the way I behave in terms of being willing to criticize or be nice, you know, how important is it? Who’s here? What’s my relationship to them?   Right now, these machines don’t have that broad social experience. And so if you know it’s going to be used for health things, a lot of reinforcement learning of the very best humans in that context would still be valuable. Eventually, the models will, having read all the literature of the world about good doctors, bad doctors, it’ll understand as soon as you say, “I want you to be a doctor diagnosing somebody.” All of the implicit reinforcement that fits that situation, you know, will be there. LEE: Yeah. GATES: And so I hope three years from now, we don’t have to do that reinforcement learning. But today, for any medical context, you would want a lot of data to reinforce tone, willingness to say things when, you know, there might be something significant at stake.  LEE: Yeah. So, you know, something Bill said, kind of, reminds me of another thing that I think we missed, which is, the context also … and the specialization also pertains to different, I guess, what we still call “modes,” although I don’t know if the idea of multimodal is the same as it was two years ago. But, you know, what do you make of all of the hubbub around—in fact, within Microsoft Research, this is a big deal, but I think we’re far from alone—you know, medical images and vision, video, proteins and molecules, cell, you know, cellular data and so on.  BUBECK: Yeah. OK. So there is a lot to say to everything … to the last, you know, couple of minutes. Maybe on the specialization aspect, you know, I think there is, hiding behind this, a really fundamental scientific question of whether eventually we have a singular AGI [artificial general intelligence] that kind of knows everything and you can just put, you know, explain your own context and it will just get it and understand everything.  That’s one vision. I have to say, I don’t particularly believe in this vision. In fact, we humans are not like that at all. I think, hopefully, we are general intelligences, yet we have to specialize a lot. And, you know, I did myself a lot of RL, reinforcement learning, on mathematics. Like, that’s what I did, you know, spent a lot of time doing that. And I didn’t improve on other aspects. You know, in fact, I probably degraded in other aspects. [LAUGHTER] So it’s … I think it’s an important example to have in mind.  LEE: I think I might disagree with you on that, though, because, like, doesn’t a model have to see both good science and bad science in order to be able to gain the ability to discern between the two?  BUBECK: Yeah, no, that absolutely. I think there is value in seeing the generality, in having a very broad base. But then you, kind of, specialize on verticals. And this is where also, you know, open-weights model, which we haven’t talked about yet, are really important because they allow you to provide this broad base to everyone. And then you can specialize on top of it.  LEE: So we have about three hours of stuff to talk about, but our time is actually running low. BUBECK: Yes, yes, yes.   LEE: So I think I want … there’s a more provocative question. It’s almost a silly question, but I need to ask it of the two of you, which is, is there a future, you know, where AI replaces doctors or replaces, you know, medical specialties that we have today? So what does the world look like, say, five years from now?  GATES: Well, it’s important to distinguish healthcare discovery activity from healthcare delivery activity. We focused mostly on delivery. I think it’s very much within the realm of possibility that the AI is not only accelerating healthcare discovery but substituting for a lot of the roles of, you know, I’m an organic chemist, or I run various types of assays. I can see those, which are, you know, testable-output-type jobs but with still very high value, I can see, you know, some replacement in those areas before the doctor.   The doctor, still understanding the human condition and long-term dialogues, you know, they’ve had a lifetime of reinforcement of that, particularly when you get into areas like mental health. So I wouldn’t say in five years, either people will choose to adopt it, but it will be profound that there’ll be this nearly free intelligence that can do follow-up, that can help you, you know, make sure you went through different possibilities.  And so I’d say, yes, we’ll have doctors, but I’d say healthcare will be massively transformed in its quality and in efficiency by AI in that time period.  LEE: Is there a comparison, useful comparison, say, between doctors and, say, programmers, computer programmers, or doctors and, I don’t know, lawyers?  GATES: Programming is another one that has, kind of, a mathematical correctness to it, you know, and so the objective function that you’re trying to reinforce to, as soon as you can understand the state machines, you can have something that’s “checkable”; that’s correct. So I think programming, you know, which is weird to say, that the machine will beat us at most programming tasks before we let it take over roles that have deep empathy, you know, physical presence and social understanding in them.  LEE: Yeah. By the way, you know, I fully expect in five years that AI will produce mathematical proofs that are checkable for validity, easily checkable, because they’ll be written in a proof-checking language like Lean or something but will be so complex that no human mathematician can understand them. I expect that to happen.   I can imagine in some fields, like cellular biology, we could have the same situation in the future because the molecular pathways, the chemistry, biochemistry of human cells or living cells is as complex as any mathematics, and so it seems possible that we may be in a state where in wet lab, we see, Oh yeah, this actually works, but no one can understand why.  BUBECK: Yeah, absolutely. I mean, I think I really agree with Bill’s distinction of the discovery and the delivery, and indeed, the discovery’s when you can check things, and at the end, there is an artifact that you can verify. You know, you can run the protocol in the wet lab and see [if you have] produced what you wanted. So I absolutely agree with that.   And in fact, you know, we don’t have to talk five years from now. I don’t know if you know, but just recently, there was a paper that was published on a scientific discovery using o3- mini (opens in new tab). So this is really amazing. And, you know, just very quickly, just so people know, it was about this statistical physics model, the frustrated Potts model, which has to do with coloring, and basically, the case of three colors, like, more than two colors was open for a long time, and o3 was able to reduce the case of three colors to two colors.   LEE: Yeah.  BUBECK: Which is just, like, astounding. And this is not … this is now. This is happening right now. So this is something that I personally didn’t expect it would happen so quickly, and it’s due to those reasoning models.   Now, on the delivery side, I would add something more to it for the reason why doctors and, in fact, lawyers and coders will remain for a long time, and it’s because we still don’t understand how those models generalize. Like, at the end of the day, we are not able to tell you when they are confronted with a really new, novel situation, whether they will work or not.  Nobody is able to give you that guarantee. And I think until we understand this generalization better, we’re not going to be willing to just let the system in the wild without human supervision.  LEE: But don’t human doctors, human specialists … so, for example, a cardiologist sees a patient in a certain way that a nephrologist …  BUBECK: Yeah. LEE: … or an endocrinologist might not. BUBECK: That’s right. But another cardiologist will understand and, kind of, expect a certain level of generalization from their peer. And this, we just don’t have it with AI models. Now, of course, you’re exactly right. That generalization is also hard for humans. Like, if you have a human trained for one task and you put them into another task, then you don’t … you often don’t know. LEE: OK. You know, the podcast is focused on what’s happened over the last two years. But now, I’d like one provocative prediction about what you think the world of AI and medicine is going to be at some point in the future. You pick your timeframe. I don’t care if it’s two years or 20 years from now, but, you know, what do you think will be different about AI in medicine in that future than today?  BUBECK: Yeah, I think the deployment is going to accelerate soon. Like, we’re really not missing very much. There is this enormous capability overhang. Like, even if progress completely stopped, with current systems, we can do a lot more than what we’re doing right now. So I think this will … this has to be realized, you know, sooner rather than later.  And I think it’s probably dependent on these benchmarks and proper evaluation and tying this with regulation. So these are things that take time in human society and for good reason. But now we already are at two years; you know, give it another two years and it should be really …   LEE: Will AI prescribe your medicines? Write your prescriptions?  BUBECK: I think yes. I think yes.  LEE: OK. Bill?  GATES: Well, I think the next two years, we’ll have massive pilots, and so the amount of use of the AI, still in a copilot-type mode, you know, we should get millions of patient visits, you know, both in general medicine and in the mental health side, as well. And I think that’s going to build up both the data and the confidence to give the AI some additional autonomy. You know, are you going to let it talk to you at night when you’re panicked about your mental health with some ability to escalate? And, you know, I’ve gone so far as to tell politicians with national health systems that if they deploy AI appropriately, that the quality of care, the overload of the doctors, the improvement in the economics will be enough that their voters will be stunned because they just don’t expect this, and, you know, they could be reelected [LAUGHTER] just on this one thing of fixing what is a very overloaded and economically challenged health system in these rich countries.  You know, my personal role is going to be to make sure that in the poorer countries, there isn’t some lag; in fact, in many cases, that we’ll be more aggressive because, you know, we’re comparing to having no access to doctors at all. And, you know, so I think whether it’s India or Africa, there’ll be lessons that are globally valuable because we need medical intelligence. And, you know, thank god AI is going to provide a lot of that.  LEE: Well, on that optimistic note, I think that’s a good way to end. Bill, Seb, really appreciate all of this.   I think the most fundamental prediction we made in the book is that AI would actually find its way into the practice of medicine, and I think that that at least has come true, maybe in different ways than we expected, but it’s come true, and I think it’ll only accelerate from here. So thanks again, both of you.  [TRANSITION MUSIC]  GATES: Yeah. Thanks, you guys.  BUBECK: Thank you, Peter. Thanks, Bill.  LEE: I just always feel such a sense of privilege to have a chance to interact and actually work with people like Bill and Sébastien.    With Bill, I’m always amazed at how practically minded he is. He’s really thinking about the nuts and bolts of what AI might be able to do for people, and his thoughts about underserved parts of the world, the idea that we might actually be able to empower people with access to expert medical knowledge, I think is both inspiring and amazing.   And then, Seb, Sébastien Bubeck, he’s just absolutely a brilliant mind. He has a really firm grip on the deep mathematics of artificial intelligence and brings that to bear in his research and development work. And where that mathematics takes him isn’t just into the nuts and bolts of algorithms but into philosophical questions about the nature of intelligence.   One of the things that Sébastien brought up was the state of evaluation of AI systems. And indeed, he was fairly critical in our conversation. But of course, the world of AI research and development is just moving so fast, and indeed, since we recorded our conversation, OpenAI, in fact, released a new evaluation metric that is directly relevant to medical applications, and that is something called HealthBench. And Microsoft Research also released a new evaluation approach or process called ADeLe.   HealthBench and ADeLe are examples of new approaches to evaluating AI models that are less about testing their knowledge and ability to pass multiple-choice exams and instead are evaluation approaches designed to assess how well AI models are able to complete tasks that actually arise every day in typical healthcare or biomedical research settings. These are examples of really important good work that speak to how well AI models work in the real world of healthcare and biomedical research and how well they can collaborate with human beings in those settings.  You know, I asked Bill and Seb to make some predictions about the future. You know, my own answer, I expect that we’re going to be able to use AI to change how we diagnose patients, change how we decide treatment options.   If you’re a doctor or a nurse and you encounter a patient, you’ll ask questions, do a physical exam, you know, call out for labs just like you do today, but then you’ll be able to engage with AI based on all of that data and just ask, you know, based on all the other people who have gone through the same experience, who have similar data, how were they diagnosed? How were they treated? What were their outcomes? And what does that mean for the patient I have right now? Some people call it the “patients like me” paradigm. And I think that’s going to become real because of AI within our lifetimes. That idea of really grounding the delivery in healthcare and medical practice through data and intelligence, I actually now don’t see any barriers to that future becoming real.  [THEME MUSIC]  I’d like to extend another big thank you to Bill and Sébastien for their time. And to our listeners, as always, it’s a pleasure to have you along for the ride. I hope you’ll join us for our remaining conversations, as well as a second coauthor roundtable with Carey and Zak.   Until next time.   [MUSIC FADES]