Cloudflare to block AI crawlers on ad pages by 2026

Cloudflare announced on July 1 that it will improve how it manages web crawlers, giving them labels like search, proxy, and training based on their behavior. By September 15, 2026, it plans to block AI agents and training crawlers from accessing ad-supported pages by default.

Cloudflare says most website owners want their content to be found by AI without losing revenue from intellectual property. Responsible AI companies clearly label their crawlers, but many mixed crawlers still exist with multiple functions. Cloudflare's crawler manager will match all behavior patterns for mixed crawlers, so if one behavior is banned, the crawler can't access that site.

Cloudflare will also offer a new attribution dashboard for website owners, adapting to the shift from SEO to GEO and AEO (Answer Engine Optimization). It will introduce page change monitoring to avoid useless repeated crawling and offer a pay-per-use crawling model.