Tech Giants Back New AI Crawling Opt-Out Standard

Google, Microsoft, Anthropic and OpenAI have announced support for a new web crawling standard that lets website operators block AI training scrapers with a single robots.txt entry, rather than adding separate entries for each provider. The proposal has been submitted to the Internet Engineering Task Force for incorporation into the Robots Exclusion Protocol. Microsoft will also offer an AI Grounding setting to block access requests tied to answering user queries, not just training. OpenAI plans similar controls. Meta expressed support but has not committed. Google plans to adopt the standard next year; Microsoft and OpenAI within months.