Cloudflare, one of the biggest network internet infrastructure companies in the world, has announced AI Labyrinth, a new tool to fight web-crawling bots that scrape sites for AI training data without permission. The company says in a blog post that when it detects inappropriate bot behavior, the free, opt-in tool lures crawlers down a path of links to AI-generated decoy pages that slow down, confuse, and waste the resources of those acting in bad faith.Websites have long used the honor system approach of robots.txt, a text file that gives or denies permission to scrapers, but which AI companies, even well-known ones like Anthropic and Perplexity AI, have been accused of ignoring. Cloudflare writes that it sees over 50 billion web crawler requests per day, and although it has tools for spotting and blocking the malicious ones, this often prompts attackers to switch tactics in a never-ending arms race.Cloudflare says rather than block bots, AI Labyrinth fights back by making them process data that has nothing to do with a given websites actual data. The company says it also functions as a next-generation honeypot, drawing in AI crawlers that keep following links to fake pages deeper, whereas a regular human being wouldnt. It says this makes it easier to fingerprint malicious bots for Cloudflares list of bad actors as well as identify new bot patterns and signatures it wouldnt have detected otherwise. According to the post, these links shouldnt be visible to human visitors.RelatedYou can read more about how AI Labyrinth works on Cloudflares blog, but heres a bit more detail from the post:We found that generating a diverse set of topics first, then creating content for each topic, produced more varied and convincing results. It is important to us that we dont generate inaccurate content that contributes to the spread of misinformation on the Internet, so the content we generate is real and related to scientific facts, just not relevant or proprietary to the site being crawled.Website administrators can opt into using AI Labyrinth by navigating to the Bot Management section of their sites Cloudflare dashboards settings and toggling it on. The company says that this is only the first iteration of using generative AI to thwart bots. It plans to create whole networks of linked URLs that bots that end up in will have a hard time clocking as fake. As Ars Technica notes, AI Labyrinth sounds similar to Nepenthes, a tool thats designed to sideline crawlers for months in a hell of AI-generated junk data.See More: