www.technologyreview.com
Earlier this month, Mark Zuckerberg announced that Meta will cut back on its content moderation efforts and eliminate fact-checking in the US in favor of the more democratic approach that X (formerly Twitter) calls Community Notes, rolling back protections that he claimed had been developed only in response to media and government pressure. The move is raising alarm bells, and rightly so. Meta has left a trail of moderation controversies in its wake, from overmoderating images of breastfeeding women to undermoderating hate speech in Myanmar, contributing to the genocide of Rohingya Muslims. Meanwhile, ending professional fact-checking creates the potential for misinformation and hate to spread unchecked. Enlisting volunteers is how moderation started on the Internet, long before social media giants realized that centralized efforts were necessary. And volunteer moderation can be successful, allowing for the development of bespoke regulations aligned with the needs of particular communities. But without significant commitment and oversight from Meta, such a system cannot contend with how much content is shared across the companys platforms, and how fast. In fact, the jury is still out on how well it works at X, which is used by 21% of Americans (Metas are significantly more popularFacebook alone is used by 70% of Americans, according to Pew). Community Notes, which started in 2021 as Birdwatch, is a community-driven moderation system on X that allows users who sign up for the program to add context to posts. Having regular users provide public fact-checking is relatively new, and so far results are mixed. For example, researchers have found that participants are more likely to challenge content they disagree with politically and that flagging content as false does not reduce engagement, but they have also found that the notes are typically accurate and can help reduce the spread of misleading posts. Im a community moderator who researches community moderation. Heres what Ive learned about the limitations of relying on volunteers for moderationand what Meta needs to do to succeed: 1. The system will miss falsehoods and could amplify hateful content There is a real risk under this style of moderation that only posts about things that a lot of people know about will get flagged in a timely manneror at all. Consider how a post with a picture of a death cap mushroom and the caption Tasty might be handled under Community Notesstyle moderation. If an expert in mycology doesnt see the post, or sees it only after its been widely shared, it may not get flagged as Poisonous, do not eatat least not until its too late. Topic areas that are more esoteric will be undermoderated. This could have serious impacts on both individuals (who may eat a poisonous mushroom) and society (if a falsehood spreads widely). Crucially, Xs Community Notes arent visible to readers when they are first added. A note becomes visible to the wider user base only when enough contributors agree that it is accurate by voting for it. And not all votes count. If a note is rated only by people who tend to agree with each other, it wont show up. X does not make a note visible until theres agreement from people who have disagreed on previous ratings. This is an attempt to reduce bias, but its not foolproof. It still relies on peoples opinions about a note and not on actual facts. Often whats needed is expertise. I moderate a community on Reddit called r/AskHistorians. Its a public history site with over 2 million members and is very strictly moderated. We see people get facts wrong all the time. Sometimes these are straightforward errors. But sometimes there is hateful content that takes experts to recognize. One time a question containing a Holocaust-denial dog whistle escaped review for hours and ended up amassing hundreds of upvotes before it was caught by an expert on our team. Hundreds of peopleprobably with very different voting patterns and very different opinions on a lot of topicsnot only missed the problematic nature of the content but chose to promote it through upvotes. This happens with answers to questions, too. People who arent experts in history will upvote outdated, truthy-sounding answers that arent actually correct. Conversely, they will downvote good answers if they reflect viewpoints that are tough to swallow.r/AskHistorians works because most of its moderators are expert historians. If Meta wants its Community Notesstyle program to work, it should make sure that the people with the knowledge to make assessments see the posts and that expertise is accounted for in voting, especially when theres a misalignment between common understanding and expert knowledge. 2. It wont work without well-supported volunteers Metas paid content moderators review the worst of the worstincluding gore, sexual abuse and exploitation, and violence. As a result, many have suffered severe trauma, leading to lawsuits and unionization efforts. When Meta cuts resources from its centralized moderation efforts, it will be increasingly up to unpaid volunteers to keep the platform safe. Community moderators dont have an easy job. On top of exposure to horrific content, as identifiable members of their communities, they are also often subject to harassment and abusesomething we experience daily on r/AskHistorians. However, community moderators moderate only what they can handle. For example, while I routinely manage hate speech and violent language, as a moderator of a text-based community I am rarely exposed to violent imagery. Community moderators also work as a team. If I do get exposed to something I find upsetting or if someone is being abusive, my colleagues take over and provide emotional support. I also care deeply about the community I moderate. Care for community, supportive colleagues, and self-selection all help keep volunteer moderators morale high(ish). Its unclear how Metas new moderation system will be structured. If volunteers choose what content they flag, will that replicate Xs problem, where partisanship affects which posts are flagged and how? Its also unclear what kind of support the platform will provide. If volunteers are exposed to content they find upsetting, will Metathe company that is currently being sued for damaging the mental health of its paid content moderatorsprovide social and psychological aid? To be successful, the company will need to ensure that volunteers have access to such resources and are able to choose the type of content they moderate (while also ensuring that this self-selection doesnt unduly influence the notes). 3. It cant work without protections and guardrails Online communities can thrive when they are run by people who deeply care about them. However, volunteers cant do it all on their own. Moderation isnt just about making decisions on whats true or false. Its also about identifying and responding to other kinds of harmful content. Zuckerbergs decision is coupled with other changes to its community standards that weaken rules around hateful content in particular. Community moderation is part of a broader ecosystem, and it becomes significantly harder to do it when that ecosystem gets poisoned by toxic content. I started moderating r/AskHistorians in 2020 as part of a research project to learn more about the behind-the-scenes experiences of volunteer moderators. While Reddit had started addressing some of the most extreme hate on its platform by occasionally banning entire communities, many communities promoting misogyny, racism, and all other forms of bigotry were permitted to thrive and grow. As a result, my early field notes are filled with examples of extreme hate speech, as well as harassment and abuse directed at moderators. It was hard to keep up with. But halfway through 2020, something happened. After a milquetoast statement about racism from CEO Steve Huffman, moderators on the site shut down their communities in protest. And to its credit, the platform listened. Reddit updated its community standards to explicitly prohibit hate speech and began to enforce the policy more actively. While hate is still an issue on Reddit, I see far less now than I did in 2020 and 2021. Community moderation needs robust support because volunteers cant do it all on their own. Its only one tool in the box. If Meta wants to ensure that its users are safe from scams, exploitation, and manipulation in addition to hate, it cannot rely solely on community fact-checking. But keeping the user base safe isnt what this decision aims to do. Its a political move to curry favor with the new administration. Meta could create the perfect community fact-checking program, but because this decision is coupled with weakening its wider moderation practices, things are going to get worse for its users rather than better. Sarah Gilbert is research director for the Citizens and Technology Lab at Cornell University.