BAR NONE AI secretly helped write California bar exam, sparking uproar A contractor used AI to create 23 out of the 171 scored multiple-choice questions. Benj Edwards – Apr 23, 2025 3:05 pm | 7 A computer-generated gavel hovers over a laptop. Credit: Getty Images A computer-generated gavel hovers over a laptop. Credit: Getty Images Story text Size Small Standard Large Width * Standard Wide Links Standard Orange * Subscribers only   Learn more On Monday, the State Bar of California revealed that it used AI to develop a portion of multiple-choice questions on its February 2025 bar exam, causing outrage among law school faculty and test takers. The admission comes after weeks of complaints about technical problems and irregularities during the exam administration, reports the Los Angeles Times. The State Bar disclosed that its psychometrician (a person skilled in administrating psychological tests), ACS Ventures, created 23 of the 171 scored multiple-choice questions with AI assistance. Another 48 questions came from a first-year law student exam, while Kaplan Exam Services developed the remaining 100 questions. The State Bar defended its practices, telling the LA Times that all questions underwent review by content validation panels and subject matter experts before the exam. "The ACS questions were developed with the assistance of AI and subsequently reviewed by content validation panels and a subject matter expert in advance of the exam," wrote State Bar Executive Director Leah Wilson in a press release. According to the LA Times, the revelation has drawn strong criticism from several legal education experts. "The debacle that was the February 2025 bar exam is worse than we imagined," said Mary Basick, assistant dean of academic skills at the University of California, Irvine School of Law. "I'm almost speechless. Having the questions drafted by non-lawyers using artificial intelligence is just unbelievable." Katie Moran, an associate professor at the University of San Francisco School of Law who specializes in bar exam preparation, called it "a staggering admission." She pointed out that the same company that drafted AI-generated questions also evaluated and approved them for use on the exam. State bar defends AI-assisted questions amid criticism Alex Chan, chair of the State Bar's Committee of Bar Examiners, noted that the California Supreme Court had urged the State Bar to explore "new technologies, such as artificial intelligence" to improve testing reliability and cost-effectiveness. However, the California Supreme Court told the LA Times that it "was unaware that AI had been used to draft any of the multiple-choice questions" until the State Bar's Monday press release. The AI disclosure follows other problems with the February exam. Test takers reported being kicked off online testing platforms, experiencing screen lag and error messages, and encountering typos and confusing questions. These issues prompted a federal lawsuit against Meazure Learning, the exam administrator, and calls for an audit of the State Bar. A unique test of dubious quality The State Bar of California switched from the National Conference of Bar Examiners' Multistate Bar Examination to its own hybrid in-person and remote testing model last year while facing a $22 million deficit. It contracted with Kaplan Exam Services for $8.25 million to create test questions. According to the LA Times, law professors Basick and Moran had previously raised concerns about the quality of the February exam questions, noting that the 50 practice questions released before the exam "still contain numerous errors" even after editing. Basick expressed additional concerns about the use of first-year law exam questions, arguing that an exam determining minimal competence to practice law requires a different standard than one assessing first-year law school learning. The State Bar plans to ask the California Supreme Court to adjust test scores for February exam takers but has resisted returning to the National Conference of Bar Examiners exams for July, citing test security concerns with remote testing options. The Committee of Bar Examiners will meet on May 5 to discuss remedies, but Chan said the State Bar is unlikely to release all 200 exam questions or return to National Conference of Bar Examiners tests soon, as nearly half of California bar applicants want to keep the remote testing option. Benj Edwards Senior AI Reporter Benj Edwards Senior AI Reporter Benj Edwards is Ars Technica's Senior AI Reporter and founder of the site's dedicated AI beat in 2022. He's also a tech historian with almost two decades of experience. In his free time, he writes and records music, collects vintage computers, and enjoys nature. He lives in Raleigh, NC. 7 Comments