How to Block AI Training Without Blocking Google

Cloudflare flipped a switch on September 15, 2026, and it broke search visibility for sites that thought they'd only blocked AI training. The new default rules judge any crawler serving two purposes by whichever setting is stricter.

Googlebot, Applebot and Bingbot all serve two purposes now, indexing pages for search and feeding that same crawl into AI training. That's the bind Cloudflare's new setting was built to fix: it lets site owners block AI training without blocking Google, Apple or Microsoft's search crawlers in the process.

Why Blocking AI Bots Started Blocking Google Too

Modern crawlers don't run separate passes for separate jobs. Googlebot crawls a page once and reuses that data for search, for AI Overviews and for model training. A rule written to stop training had no way to leave search alone.

That overlap is exactly why showing up inside an AI-generated answer now depends on settings most site owners never thought about.

Cloudflare's own numbers show how many walked into this without noticing: seventeen percent of sites on its network had turned on some form of AI training block, while almost none had gone the other way and blocked search crawlers on purpose.

Anyone in that group who hadn't separated the two settings before September 15 lost Googlebot along with the training bots.

How to Block AI Training Without Blocking Google

Cloudflare's fix is a setting called Disallow AI Training. For Google, it writes a Disallow rule aimed at the Google-Extended token into robots.txt rather than blocking Googlebot itself. Apple gets the same split through Applebot-Extended.

Bing has no equivalent yet, since Microsoft isn't expected to support one until early 2027.

Cloudflare calls the crawlers that qualify for this split "Accountable," and only a handful have earned it: Google, Apple and Microsoft directly, plus Amazon, Anthropic, Meta and OpenAI, which already run separate crawlers for search and training.

What Site Owners Should Check Now

Photo by Tom Fisk on Pexels

Existing customers aren't left to sort this out alone. Sites that already had the legacy "Block AI bots" toggle on were migrated automatically to Disallow AI Training, so most don't need to touch anything.

New sites get a default based on whether they run ads: ad-free sites default to allowing training and AI agents, ad-supported ones default to Disallow AI Training with agents blocked on ad pages.

Still worth checking your own settings, especially if:

• Your site sits on Cloudflare's free plan and nobody has touched the AI bot settings before.
• You run ads and rely on an AI agent integration that needs those pages crawled.
• You're counting on Bing, since its no-training robots.txt support isn't live until early 2027.

The stakes reach past one settings page. Cloudflare has found that people who see an AI summary in search results are over 40% more likely to walk away without clicking any site, and content signals in robots.txt are requests a crawler operator agrees to honor, not a technical block.

Traffic that does arrive from AI-driven search converts well above a typical search visitor, so losing Googlebot on top of that shift stacks two problems onto one revenue line.

That's the same tension behind the questions courts are still working out over AI training on copyrighted content: a robots.txt line states a preference, and whether it's honored depends on who's reading it.

Working out how search visibility and organic growth connect, alongside the technical setup our AI and automation team handles, is worth doing before a default setting decides it for you.

Check your Cloudflare settings this week. The default that applies if you don't isn't necessarily the one your site actually wants.

Cover photo by Google DeepMind on Pexels