![\"Banner](\"https://www.artificialintelligence-news.com/wp-content/uploads/2025/08/ai-expo-banner-2025.png\")
AI agent crawlers, the bots that fetch pages in real time on behalf of a person waiting for an answer, will be blocked by default on a slice of the web from September 15 onwards. Cloudflare announced the change on July 1, and most of the coverage since then has focused on Google. The more useful part […]
\nThe post AI agent crawlers now need permission. Here’s how to get it appeared first on AI News.
", "content_html": "AI agent crawlers, the bots that fetch pages in real time on behalf of a person waiting for an answer, will be blocked by default on a slice of the web from September 15 onwards. Cloudflare announced the change on July 1, and most of the coverage since then has focused on Google. The more useful part is what it asks of everyone building agents, and what it offers them in return.
\n\n\n\nCloudflare has replaced its single block-AI-bots switch with three categories. Search covers bots that index a page to answer questions about it later. Agent covers automated systems acting in real time for a user, including ChatGPT’s fetch bot and browser-driving agents. Training covers crawlers that pull content into a model’s weights. The controls went live on July 1 for every customer, including the free tier.
\n\n\n\nFrom September 15, the defaults change. Training and Agent will be blocked on pages that display ads, while Search will stay allowed. The new defaults apply to domains newly onboarding to Cloudflare, new sites set up by existing customers, and all existing free-tier customers.
\n\n\n\nAnyone who does not want them can opt out through their security settings before the date.
\n\n\n\nCloudflare’s logic is that an advertisement is evidence that a page was built for a human to land on. A search crawler that sends a reader back is a referral. A bot that reads the page and hands the answer to someone else is not.
\n\n\n\nAgentic deployments have been built on the assumption that the open web stays open. A research agent fetches a competitor’s pricing page. A monitoring tool checks a supplier’s announcements. A customer-service agent pulls a manufacturer’s specification sheet. None of this involves a licence, and until now, none of it needed one.
\n\n\n\nCloudflare sits in front of a large share of the world’s web traffic, and its blocks operate at the network level rather than as a robots.txt suggestion a crawler can ignore. Ad-supported pages are exactly the pages agents want, because that is where news, reviews, pricing and product coverage live. The failure mode for an enterprise agent is not a lawsuit. It is silence, or an answer built from whatever it could still reach.
\n\n\n\nThere is a Google-shaped complication. Googlebot crawls for both search and training in a single bot, so under the most restrictive rule, a site that blocks Training also blocks Googlebot. Cloudflare CEO Matthew Prince said the company hopes the changes will “encourage mixed-use crawlers to separate search from agent use and training”, which is a polite way of saying the pressure is the point.
\n\n\n\nAnyone running agents should start by working out which of their Cloudflare accounts will read as Agent-class. The classification is behavioural rather than something you opt into, so a research agent that browses in real time is caught whether or not its operator thinks of it as a crawler.
\n\n\n\nExpect degraded coverage rather than a clean failure, because the block lands on ad-supported pages and leaves the rest reachable. Negotiated access, not a rewritten user-agent string, is the way through.
\n\n\n\nPublishers have a different homework list. Check your tier first, since existing free-tier customers are moved to the new defaults automatically on September 15, a detail most coverage skipped. Then decide whether blocking Training is worth what it costs, because it takes Googlebot with it and your search visibility along with it.
\n\n\n\nThe mechanism worth watching is the money. Pay Per Crawl is becoming Pay Per Use, with Ceramic.ai paying publishers when their content appears in AI search results, and You.com paying when an agent reaches premium content. Cloudflare says more than half of AI crawler traffic is spent re-fetching pages that have not changed, so there is waste on both sides worth pricing out.
\n\n\n\nThis is the first round of the content fight, where the answer on offer is a rate rather than a wall.
\n\n\n\nOne weakness lies in the taxonomy itself. Search, Agent and Training are behaviours the AI companies declare about their own bots, and a firm that would rather not have its training run classified as training has an obvious incentive. The announcement does not explain what stops it.
\n\n\n\nAccess to the open web has been free and unlimited for thirty years, and the bill is now itemised. Agent builders who sort out their access before September have a workable problem. The ones who find out from a 403 will be rebuilding on the fly.
\n\n\n\nSee also: Visa ChatGPT integration enables AI agent retail purchasing
\n\n\n\nWant to learn more about AI and big data from industry leaders? Check out AI & Big Data Expo taking place in Amsterdam, California, and London. The comprehensive event is part of TechEx and is co-located with other leading technology events including the Cyber Security & Cloud Expo. Click here for more information.
\n\n\n\nAI News is powered by TechForge Media. Explore other upcoming enterprise technology events and webinars here.
\nThe post AI agent crawlers now need permission. Here’s how to get it appeared first on AI News.
", "content_text": "AI agent crawlers, the bots that fetch pages in real time on behalf of a person waiting for an answer, will be blocked by default on a slice of the web from September 15 onwards. Cloudflare\nannounced\nthe change on July 1, and most of the coverage since then has focused on Google. The more useful part is what it asks of everyone building agents, and what it offers them in return.\nCloudflare has replaced its single block-AI-bots switch with three categories. Search covers bots that index a page to answer questions about it later. Agent covers automated systems acting in real time for a user, including ChatGPT’s fetch bot and browser-driving agents. Training covers crawlers that pull content into a model’s weights. The controls went live on July 1 for every customer, including the free tier.\nFrom September 15, the defaults change. Training and Agent will be blocked on pages that display ads, while Search will stay allowed. The new defaults apply to domains newly onboarding to Cloudflare, new sites set up by existing customers, and all existing free-tier customers.\nAnyone who does not want them can opt out through their security settings before the date.\nCloudflare’s logic is that an advertisement is evidence that a page was built for a human to land on. A search crawler that sends a reader back is a referral. A bot that reads the page and hands the answer to someone else is not.\nWhat AI agent crawlers run into now\nAgentic deployments have been built on the assumption that the open web stays open. A research agent fetches a competitor’s pricing page. A monitoring tool checks a supplier’s announcements. A customer-service agent pulls a manufacturer’s specification sheet. None of this involves a licence, and until now, none of it needed one.\nCloudflare sits in front of a large share of the world’s web traffic, and its blocks operate at the network level rather than as a robots.txt suggestion a crawler can ignore. Ad-supported pages are exactly the pages agents want, because that is where news, reviews, pricing and product coverage live. The failure mode for an enterprise agent is not a lawsuit. It is silence, or an answer built from whatever it could still reach.\nThere is a Google-shaped complication. Googlebot crawls for both search and training in a single bot, so under the most restrictive rule, a site that blocks Training also blocks Googlebot. Cloudflare CEO Matthew Prince said the company hopes the changes will “encourage mixed-use crawlers to separate search from agent use and training”, which is a polite way of saying the pressure is the point.\nGetting permission for AI agent crawlers\nAnyone running agents should start by working out which of their Cloudflare accounts will read as Agent-class. The classification is behavioural rather than something you opt into, so a research agent that browses in real time is caught whether or not its operator thinks of it as a crawler.\nExpect degraded coverage rather than a clean failure, because the block lands on ad-supported pages and leaves the rest reachable. Negotiated access, not a rewritten user-agent string, is the way through.\nPublishers have a different homework list. Check your tier first, since existing free-tier customers are moved to the new defaults automatically on September 15, a detail most coverage skipped. Then decide whether blocking Training is worth what it costs, because it takes Googlebot with it and your search visibility along with it.\nThe mechanism worth watching is the money. Pay Per Crawl is becoming Pay Per Use, with Ceramic.ai paying publishers when their content appears in AI search results, and You.com paying when an agent reaches premium content. Cloudflare says more than half of AI crawler traffic is spent re-fetching pages that have not changed, so there is waste on both sides worth pricing out.\nThis is the first round of the content fight, where the answer on offer is a rate rather than a wall.\nOne weakness lies in the taxonomy itself. Search, Agent and Training are behaviours the AI companies declare about their own bots, and a firm that would rather not have its training run classified as training has an obvious incentive. The announcement does not explain what stops it.\nAccess to the open web has been free and unlimited for thirty years, and the bill is now itemised. Agent builders who sort out their access before September have a workable problem. The ones who find out from a 403 will be rebuilding on the fly.\nSee also:\nVisa ChatGPT integration enables AI agent retail purchasing\nWant to learn more about AI and big data from industry leaders?\nCheck out\nAI & Big Data Expo\ntaking place in Amsterdam, California, and London. The comprehensive event is part of\nTechEx\nand is co-located with other leading technology events including the\nCyber Security & Cloud Expo\n. Click\nhere\nfor more information.\nAI News is powered by\nTechForge Media\n. Explore other upcoming enterprise technology events and webinars\nhere\n.\nThe post\nAI agent crawlers now need permission. Here’s how to get it\nappeared first on\nAI News\n.", "image": { "url": "https://virtues.guru/tech-assets/45_Semiconductor_outlines.jpg", "file": "45_Semiconductor_outlines.jpg", "title": "File:Semiconductor outlines.jpg" }, "language": "en", "translations": { "pt": { "title": "Agentes de IA agora precisam de permissão. Veja como obtê-la", "content_text": "A Cloudflare bloqueará os crawlers de agentes de IA por padrão em páginas com publicidade a partir de 15 de setembro. A mudança afeta três categorias: Busca, Agente e Treinamento. Os editores devem decidir se bloquearão o Treinamento, o que também bloqueará o Googlebot. Os criadores de agentes devem classificar suas contas como de \"classe Agente\" e negociar o acesso. A lógica da Cloudflare é que a publicidade indica que uma página é para humanos, e não para bots.", "summary_en": "Cloudflare will block AI agent crawlers by default on ad-supported pages from September 15. The change affects three categories: Search, Agent, and Training. Publishers must decide whether to block Training, which also blocks Googlebot. Agent builders must classify their accounts as Agent-class and negotiate access. Cloudflare's logic is that ads indicate a page is for humans, not bots.", "translated_at": "2026-07-14T06:23:03.498020+00:00", "model": "translategemma:4b", "summary_model": "llama3.1:8b", "source": "json_summary" } } }