Cloudflare AI crawler block: is your site now hidden from AI search?

If your website runs on Cloudflare's free plan, or you have set up a new site on Cloudflare recently, its default AI settings changed on 15 September. The Cloudflare AI crawler block now stops "mixed-use" crawlers — bots that combine search indexing with AI training or agent use — from reaching pages that carry ads, unless you change the setting. For publishers that may be exactly what they want. For businesses that want to be found in AI search and assistants, it is worth checking what your site now allows.

What changed on 15 September

Cloudflare announced the change in July. As TechCrunch reported, starting 15 September 2026, Cloudflare's default settings block mixed-use crawlers from pages that host ads. The new defaults apply to:

• new Cloudflare customers;
• new sites set up by existing customers;
• all existing free-plan customers.

Site owners can change the setting. Cloudflare's aim, in CEO Matthew Prince's words, is to "encourage mixed-use crawlers to separate out search from agent use and training". Cloudflare has singled out the world's largest search engine — widely read as Google — as having an advantage because it is hard to remain in search without also feeding AI. Google points to its separate Google-Extended control, which lets sites opt out of AI training without affecting Search, though Googlebot itself also powers AI features in Search.

Cloudflare AI crawler block: who is affected

The default applies to three groups — new Cloudflare customers, new sites added by existing customers, and every existing free-plan customer. Paid-plan sites that were already set up keep their current configuration unless the owner changes it. Within an affected site, the block applies to pages that host ads, so an ad-free corporate site, a SaaS dashboard or a documentation site may see little difference, while a content site monetised with ads will see mixed-use crawlers turned away.

Many small businesses run on the free plan without anyone actively managing Cloudflare — the account was set up once by a developer or agency. Those are the sites most likely to have changed without anyone noticing.

Pay Per Crawl becomes Pay Per Use

Alongside the default change, Cloudflare is turning its Pay Per Crawl marketplace into "Pay Per Use": publishers get paid when their content actually creates value for an AI application — for example appearing in a partner's AI search results — rather than per page fetched. Ceramic.ai and You.com are the launch partners. Cloudflare also says more than half of AI crawler traffic is spent re-fetching pages that have not changed.

Is Cloudflare blocking AI crawlers on my site?

Possibly. Check your Cloudflare dashboard's AI crawler settings (AI Crawl Control) for each zone, especially if you are on the free plan or created the site recently. Remember the rule applies to pages that host ads; a site without ads may see little change. As one detailed explainer notes, the dashboard can make the impact look larger than it is, so look at which crawlers are actually being blocked on which pages rather than the headline numbers.

Should you allow or block AI crawlers?

It depends on what your site is for.

• Publishers and content businesses whose content is the product often benefit from blocking or charging — the new defaults and Pay Per Use are built for them.
• Service businesses, SaaS companies and agencies usually want the opposite: to be cited when someone asks an AI assistant for a recommendation. Blocking the crawlers that feed those answers can quietly remove you from them.
• E-commerce sits in between: product pages want discovery, while proprietary content such as buying guides may deserve protection.

The decision should be deliberate, per crawler and per section of the site, not left to a default. Our guide to GPTBot and OAI-SearchBot controls explains how the main AI crawlers differ.

How to allow AI search crawlers on Cloudflare

1. Inventory your zones. List every domain on Cloudflare and its plan.
2. Review AI crawler settings for each, noting what is blocked by default.
3. Decide per crawler category: search, AI search and assistants, and training. Allow what brings you customers; block or charge what does not.
4. Align robots.txt with your Cloudflare settings so the two do not contradict each other.
5. Monitor results. Watch crawler logs and referrals from AI assistants over the following weeks.

What it means for agent builders

If you build AI agents or tools that fetch web pages, expect more blocks. Identify your crawler honestly with a clear user agent, separate search from training traffic, respect robots.txt, and consider structured alternatives: official APIs, feeds, or site-provided tools such as WebMCP, which let sites offer agents a sanctioned interface instead of scraping.

Will blocking AI crawlers hurt SEO?

Blocking crawlers used only for AI training does not affect traditional Google rankings. Blocking crawlers that power AI search features can reduce how often your site appears in AI-generated answers, which increasingly sit above the classic results. That is the trade-off to decide consciously.

If you manage several sites on Cloudflare and want a consistent policy — which AI crawlers to allow, block or charge — our DevOps and cloud engineering team can audit the settings and set them up to match your business goals.