Major AI firms adopt new standard letting sites block AI scraping

Websites have long used the robots.txt file to tell crawlers what they can and cannot fetch. As AI crawlers keep scraping web content for model training, publishers have had to set their preferences with each AI company one by one, which is a heavy burden.

OpenAI, Google, Microsoft and Anthropic recently announced a new standard. Publishers can now state their content preferences for AI training to many AI companies at once, instead of setting rules one by one. The standard was proposed by Anthropic tech lead Kevin Kelley to the IETF and will be built into robots.txt.

Anthropic has adopted it first. OpenAI plans to roll it out in the coming months, while Google and Microsoft expect to support it next year. Meta's spokesperson also voiced support, but gave no specific timeline. Once live, publishers will manage content permissions much more easily.