Cloudflare AI Crawler Defaults Change September 15: Your Client Audit Plan

Cloudflare set a date that is going to change how your clients think about SEO and AI visibility in the same breath. Starting September 15, 2026, Cloudflare will block Training and Agent AI crawlers by default on any ad-supported page, across the roughly one in five domains that sit behind its network.

If your agency has clients on Cloudflare who run display ads, sponsored placements, or affiliate programs, the Cloudflare AI crawler policy change lands in seven weeks and it needs a call this month. Two weeks after that, you also have the Google Ads Target CPA behavior change on the horizon. Stack these correctly and your clients get ahead of both shifts before Q4 planning.

What the Cloudflare AI crawler policy actually says

The new default applies to new domains, new sites added by existing customers, and all free-tier customers. Existing paid domains keep their current configuration unless the site owner explicitly opts in. On ad-supported pages, three separate categories now have their own switches: Search stays allowed by default, Agent gets blocked by default, and Training gets blocked by default.

The critical wrinkle is what Cloudflare calls the most restrictive applicable rule. If a crawler mixes purposes and one of those purposes is Training, the entire crawler gets blocked when Training is blocked. Cloudflare’s own AI Crawl Control documentation spells out how each category is enforced, and it names Googlebot, Applebot, and BingBot as multi-purpose crawlers that fall under this rule.

Put another way: if your client site is behind Cloudflare on a free plan, runs display ads, and does nothing before September 15, there is a real path where Googlebot gets blocked from your ad pages. That is not a hypothetical. It is what the most restrictive applicable rule produces when Training is default-off.

Why the Cloudflare AI crawler change matters for agency clients

Ad-supported publishers were the loudest voice pushing for this policy. But the definition of an ad-supported page is broad. Any site running Google AdSense, display network ads, affiliate widgets that render on the page, or programmatic units gets pulled into the new default. That covers a lot of your smaller ecommerce clients, most of your affiliate marketing accounts, and every publisher client you have.

The TechRadar analysis of the Cloudflare policy shift laid out the worst-case reading in one sentence: every website with ads currently using Cloudflare could end up completely blocked from Google search. That is only the worst case, and existing paid customers keep their prior configuration. But new domains and free-tier accounts land in the strict posture unless someone flips the switches back.

Layer this onto the trend your team already tracked in our publishers blocking Google AI Overviews playbook. Publishers spent Q2 deciding whether to opt out of AI training. Cloudflare has now made that decision the default for new zones. The industry-wide direction of travel is toward stricter, more granular control at the network edge.

Your Cloudflare AI crawler audit for every client

Skip the deep architectural review. Here is what your senior strategist should do this week for each client behind Cloudflare.

1. Log into the Cloudflare dashboard for each client, open Security, then AI Crawl Control. Note when the zone was added. Any zone added after July 2025 already starts with unverified AI crawlers blocked. Zones added before that date inherit whatever the last operator set.
2. Under AI Crawl Control, open the Crawlers tab. Screenshot the current Allow or Block state for GPTBot, ClaudeBot, PerplexityBot, OAI-SearchBot, ChatGPT-User, Googlebot, Applebot, and BingBot. This is your baseline before September 15.
3. Confirm at the edge, not just in the dashboard. Run curl with each crawler’s user agent against the client’s production URL and read the HTTP status code back. A 200 means the request passes. A 403 or 429 means Cloudflare is blocking. The dashboard sometimes shows a rule that the edge does not fully enforce, so trust the curl output.
4. Decide per client whether Agent traffic is a threat or a sales channel. For ecommerce clients where ChatGPT-User visits often precede a purchase, Agent is a sales channel and should stay allowed. For content clients paid by display impressions, Agent is a threat because a bot loading the page never sees the ad.

Document the Cloudflare AI crawler decision in your client folder with a specific date. This becomes the evidence that you managed the transition rather than discovering it in October traffic reports.

Once the audit is done, sort each client into one of three profiles. Not every client needs the same Cloudflare AI crawler setup, and three quick profiles cover most of what your agency will see.

Ecommerce sites with light display ads or affiliate widgets. Keep Search allowed. Allow Agent because a real user asked ChatGPT to fetch the page. Block Training because letting models train on your product catalog gives away a moat with no return. If you are a Cloudflare paid customer, make this explicit in Security settings rather than relying on defaults.

Content and publisher sites with heavy ad revenue. Keep Search allowed. Block Training. Block Agent, because an agent that reads your article and summarizes it upstream never fires an ad impression. This is the profile Cloudflare tuned the new default for, so paid customers can leave the default alone if they add a new domain after September 15.

Lead-generation and B2B service sites that want AI visibility. Keep Search allowed. Keep Agent allowed because being reachable inside ChatGPT and Perplexity is a lead source you cannot buy with paid media. Block Training on a case-by-case basis, understanding that if Googlebot bundles Search and Training you may need to opt out of the most restrictive rule for that specific crawler.

Our GA4 Source Group reporting playbook from yesterday is the measurement side of the same conversation. If you are going to make Cloudflare decisions per client, you need to also see the AI referral traffic those decisions affect. Do the GA4 Source Group swap first, then have the Cloudflare conversation with the same numbers in hand.

Where this lands in your Q3 client roadmap

Your team is already juggling several measurement and policy changes for Q3. The Google Ads Target CPA change on August 17 shifts how budget-limited campaigns behave. The EU AI Act Article 50 content labeling deadline hits August 2. The Google Ads API v25 lifecycle goal rewrite is still working through your engineering pipeline.

Sequence the work like this. Handle the EU AI Act labeling by August 2. Handle the Google Ads Target CPA review by August 17. Then reserve the last three weeks of August for the Cloudflare AI crawler audits, so you finish before September 15 with room to test.

The recent Google search ranking volatility playbook is your reference on the SEO measurement side. Between the July volatility signal and the September Cloudflare defaults, your organic reporting will look different by October than it did in June. Set the client expectation now, in the language of specific dates, so the movement lands as managed change rather than a surprise.

What the Cloudflare AI crawler shift does not fix

The new controls only work at the Cloudflare edge. If your client has content mirrored on a subdomain that does not sit behind Cloudflare, none of this policy touches that traffic. Audit the DNS records and confirm which hostnames actually route through Cloudflare before you promise anything to the client.

Verified bot status is honesty-based, not proof-based. A crawler declares its category and Cloudflare enforces the label. If a crawler misrepresents itself or routes through residential IPs, the new default does not catch it. The open-source AI agent tools coverage we published earlier this summer covers the harder edge case: agents that scrape without honest identification.

Analytics reconciliation is going to be messy for two months after September 15. Some AI referral traffic that used to show up will stop. Some spam that inflated numbers will also stop. Do not confuse a signal for a problem or a problem for a signal. Baseline everything before the change so you have something to compare against.

Our Search Console platform properties post covers the parallel measurement update on the Search Console side. Between platform properties in GSC, Source Group in GA4, and the Cloudflare AI crawler controls, your organic and AI-referral reporting has three native levers that did not exist in Q1.

Book two hours next Friday for the seven-week action plan. Pull a list of every client on Cloudflare from your operations tracker. Grade each client by ad-supported yes or no, and by whether AI visibility is a sales channel. Send that grid to the account managers as a decision worksheet.

Any account graded ad-supported yes and AI visibility yes gets a one-hour senior review before September 15. Any account graded ad-supported no can wait until Q4. Any account graded ad-supported yes and AI visibility no can be handled by the account manager in a fifteen-minute settings review.

If a client is unsure about the AI creative side of the conversation, our Claude Opus 5 for marketing creative guide is a useful adjacent read. The Cloudflare change touches how AI companies access content. The creative side is what your client makes with the AI once it has that content.

If you want a second set of eyes on which client accounts sit in the biggest risk quadrant, book a free consultation and our team will walk the grid with you.

Let’s Grow!