Apple Intelligence Apple will update iOS notification summaries after BBC headline mistake It's unfortunately not possible for Apple Intelligence to make zero errors. Samuel Axon Jan 6, 2025 7:28 pm | 35 Apple Intelligence was unveiled at WWDC 2024. Credit: Apple Apple Intelligence was unveiled at WWDC 2024. Credit: Apple Story textSizeSmallStandardLargeWidth *StandardWideLinksStandardOrange* Subscribers only Learn moreApple plans to release a software update that is meant to help users understand better that its notification summaries are AI-generated and may contain errors, according to a recent BBC news story. The update is a response to reports that the summaries gave users misleading information about world events.For example, one false summary suggested to at least one user that Luigi Mangione, the alleged murderer of UnitedHealthcare CEO Brian Thompson, had committed suicide. The notification in question was meant to summarize the most important tidbits from 22 BBC news app notifications, according to a widely circulated screenshot.Apple hasn't publicly specified exactly what will be changed to better inform usersonly that it will be a software change that will "further clarify" when the notifications have been generated via the feature that resides under Apple Intelligence umbrella.The notification summaries, which are generated for individual users and devices via a custom large language model (LLM) Apple has trained, already had a small icon on them to indicate that they were not normal, unaltered notifications from the apps in question.However, that icon's meaning may not be immediately apparent to most users.Users can disable the Apple Intelligence notification summaries by accessing the Settings app on their mobile devices.When contacted for a statement, an Apple spokesperson told Ars:Apple Intelligence is designed to help users get everyday tasks done faster and more easily. This includes optional notification summaries, which provide users who choose to opt in a way to briefly view information from apps and tap into the full details whenever they choose. These are identified by a summarization icon, and the original content is a quick tap away. Apple Intelligence features are in beta and we are continuously making improvements with the help of user feedback. A software update in the coming weeks will further clarify when the text being displayed is summarization provided by Apple Intelligence. We encourage users to report a concern if they view an unexpected notification summary.This issue cant just be fixed entirelyThe BBC stories about the error-prone AI have often seemed to lack understanding of how the Apple Intelligence notification summaries work, saying that they came "from within Apple's app" (the summaries are a system-wide feature) and suggesting that all users received the offending notification about Mangione (the wording of the summaries varies on individual devices depending on what other notifications were received around the same time).Nevertheless, it's a serious problem when the summaries misrepresent news headlines, and edge cases where this occurs are unfortunately inevitable. Apple cannot simply fix these summaries with a software update. The only answers are either to help users understand the drawbacks of the technology so they can make better-informed judgments or to remove or disable the feature completely. Apple is apparently going for the former.We're oversimplifying a bit here, but generally, LLMs like those used for Apple's notification summaries work by predicting portions of words based on what came before and are not capable of truly understanding the content they're summarizing.Further, these predictions are known to not be accurate all the time, with incorrect results occurring a few times per 100 or 1,000 outputs. As the models are trained and improvements are made, the error percentage may be reduced, but it never reaches zero when countless summaries are being produced every day.Deploying this technology at scale without users (or even the BBC, it seems) really understanding how it works is risky at best, whether it's with the iPhone's summaries of news headlines in notifications or Google's AI summaries at the top of search engine results pages. Even if the vast majority of summaries are perfectly accurate, there will always be some users who see inaccurate information.These summaries are read by so many millions of people that the scale of errors will always be a problem, almost no matter how comparatively accurate the models get.We wrote at length a few weeks ago about how the Apple Intelligence rollout seemed rushed, counter to Apple's usual focus on quality and user experience. However, with current technology, there is no amount of refinement to this feature that Apple could have done to reach a zero percent error rate with these notification summaries.We'll see how well Apple does making its users understand that the summaries may be wrong, but making all iPhone users truly grok how and why the feature works this way would be a tall order.Samuel AxonSenior EditorSamuel AxonSenior Editor Samuel Axon is a senior editor at Ars Technica. He covers Apple, software development, gaming, AI, entertainment, and mixed reality. He has been writing about gaming and technology for nearly two decades at Engadget, PC World, Mashable, Vice, Polygon, Wired, and others. He previously ran a marketing and PR agency in the gaming industry, led editorial for the TV network CBS, and worked on social media marketing strategy for Samsung Mobile at the creative agency SPCSHP. He also is an independent software and game developer for iOS, Windows, and other platforms, and heis a graduate of DePaul University, where he studied interactive media and software development. 35 Comments