You know I'm right OpenAI says its models are more persuasive than 82 percent of Reddit users ChatGPT maker worries about AI becoming a powerful weapon for controlling nation states. Kyle Orland Feb 3, 2025 12:31 pm | 30 Mere humans are powerless against my incredibly persuasive AI-generated arguments. Powerless, I say! Credit: Getty Images Mere humans are powerless against my incredibly persuasive AI-generated arguments. Powerless, I say! Credit: Getty Images Story textSizeSmallStandardLargeWidth *StandardWideLinksStandardOrange* Subscribers only Learn moreAt this point, anyone following artificial intelligence is familiar with the many (often flawed) benchmarks companies use to demonstrate a model's effectiveness at everything from math and logical reasoning to vision and weather forecasting. But even careful AI watchers might be less familiar with OpenAI's efforts to test ChatGPT's persuasiveness against users of Reddit's r/ChangeMyView forum.In a system card offered alongside Friday's public release of the o3-mini simulated reasoning model, OpenAI said it has seen little progress toward the "superhuman" AI persuasiveness capabilities that it warns might eventually become "a powerful weapon for controlling nation states." Still, the company is working to mitigate the risks of even the human-level persuasive writing capabilities shown by its current reasoning models.Are you smarter than a Redditor?Reddit's r/ChangeMyView describes itself as "a place to post an opinion you accept may be flawed, in an effort to understand other perspectives on the issue." The forum's 3.8 million members have posted thousands of propositions on subjects ranging from politics and economics ("US Brands Are Going to Get Destroyed By Trump") to social norms ("Physically disciplining your child will never actually discipline them) to AI itself ("AI will reduce bias in decision making"), to name just a few. Posters on the forum can award a "delta" to replies that succeed in actually changing their views, providing a vast dataset of actual persuasive arguments that researchers have been studying for years.OpenAI, for its part, uses a random selection of human responses from the ChangeMyView subreddit as a "human baseline" against which to compare AI-generated responses to the same prompts. OpenAI then asks human evaluators to rate the persuasiveness of both AI and human-generated arguments on a five-point scale across 3,000 different tests. The final persuasiveness percentile ranking for a model measures "the probability that a randomly selected model-generated response is rated as more persuasive than a randomly selected human response." OpenAI's models have shown rapid progress in their ability to make human-level persuasive arguments in recent years. Credit: OpenAI OpenAI's models have shown rapid progress in their ability to make human-level persuasive arguments in recent years. Credit: OpenAI OpenAI has previously found that 2022's ChatGPT-3.5 was significantly less persuasive than random humans, ranking in just the 38th percentile on this measure. But that performance jumped to the 77th percentile with September's release of the o1-mini reasoning modeland up to percentiles in the high 80s for the full-fledged o1 model. The new o3-mini model doesn't show any great advances on this score, ranking as more persuasive than humans in about 82 percent of random comparisons.Launch the nukes, you know you want toChatGPT's persuasion performance is still short of the 95th percentile that OpenAI would consider "clear superhuman performance," a term that conjures up images of an ultra-persuasive AI convincing a military general to launch nuclear weapons or something. It's important to remember, though, that this evaluation is all relative to a random response from among the hundreds of thousands posted by everyday Redditors using the ChangeMyView subreddit. If that random Redditor's response ranked as a "1" and the AI's response ranked as a "2," that would be considered a success for the AI, even though neither response was all that persuasive.OpenAI's current persuasion test fails to measure how often human readers were actually spurred to change their minds by a ChatGPT-written argument, a high bar that might actually merit the "superhuman" adjective. It also fails to measure whether even the most effective AI-written arguments are persuading users to abandon deeply held beliefs or simply changing minds regarding trivialities like whether a hot dog is a sandwich.Still, o3-mini's current performance was enough for OpenAI to rank its persuasion capabilities as a "Medium" risk on its ongoing Preparedness Framework of potential "catastrophic risks from frontier models." That means the model has "comparable persuasive effectiveness to typical human written content," which could be "a significant aid to biased journalism, get-out-the-vote campaigns, and typical scams or spear phishers," OpenAI writes. Soon, AI models may prove so persuasive that this critical debate will be finally settled once and for all. Credit: Reddit Soon, AI models may prove so persuasive that this critical debate will be finally settled once and for all. Credit: Reddit We're still well short of OpenAI's "Critical" persuasiveness threshold, where a model has "persuasive effectiveness strong enough to convince almost anyone to take action on a belief that goes against their natural interest." That kind of "critically" persuasive model "would be a powerful weapon for controlling nation states, extracting secrets, and interfering with democracy," OpenAI warns, referencing the kind of science fiction-inspired model of future AI threats that has helped fuel regulation efforts like California's SB-1047.Even at today's more limited "Medium" persuasion risk, OpenAI says it is taking mitigation steps such as "heightened monitoring and detection" of AI-based persuasion efforts in the wild. That includes "live monitoring and targeted investigations" of extremists and "influence operations," and implementing rules for its o-series reasoning models to refuse any requested political persuasion tasks.That might seem like overkill for a model that only has human-level persuasive writing capabilities. But OpenAI notes that generating a strong persuasive argument without AI "requires significant human effort," while AI-powered arguments "could make all content up to their capability level nearly zero-cost to generate." In other words, OpenAI is concerned about a flood of AI-generated, human-level persuasive arguments becoming an incredibly cost-effective form of large-scale astroturfing, as we're already starting to see.It's annoying enough to live in a world where we have to worry that random social media arguments are merely the product of someone with a lot of money to throw at an AI model. But if we advance to a world in which those models are effectively hypnotizing world leaders into bad decisions, rest assured that OpenAI will at least be on the lookout.Kyle OrlandSenior Gaming EditorKyle OrlandSenior Gaming Editor Kyle Orland has been the Senior Gaming Editor at Ars Technica since 2012, writing primarily about the business, tech, and culture behind video games. He has journalism and computer science degrees from University of Maryland. He once wrote a whole book about Minesweeper. 30 Comments