Block AI Training Crawlers Without Blocking Googlebot

If you want to keep aggressive AI scrapers from eating up your server bandwidth and lifting your content, you really need a two-part defense: one layer on your origin server and another out at your edge network. Just dropping a standard robots.txt file on your server leaves the door wide open for rogue bots that don't play by the rules. On the flip side, if you start throwing around blanket IP or User-Agent bans in your firewall, you run a very real risk of accidentally blocking Google Search, and tanking the organic traffic your site relies on.

• Prerequisites: Write access to your origin server's root directory (for robots.txt), access to configure Cloudflare WAF (or a similar CDN), and a local terminal ready to go.
• Primary Targets (AI Training): GPTBot, CCBot (Common Crawl), Anthropic-ai, Google-Extended, Cohere-ai.
• Safe List (Organic Search): Googlebot, Bingbot, Slurp.
• Execution Time: Around 15 minutes to get the policy deployed and wait for the WAF rules to kick in.

The Difference Between Googlebot and Google-Extended

Before you start blocking things, you need to understand how Google splits up its crawling operations. If you block the main Googlebot token, you will immediately wipe your site out of Google Search and disappear from AI Overviews.

When Google scrapes the web specifically to train its foundation models (like Gemini and the Vertex AI APIs), it uses a completely different user agent token: Google-Extended. Blocking Google-Extended in your robots.txt or WAF tells Google not to use your content for generative AI training, without touching your traditional search rankings, indexing speed, or organic visibility. The two operate completely independently of each other.

Classifying Crawlers: Search, AI Search, and AI Training

Before you start writing rules, it helps to sort the bots hitting your site into three distinct buckets. Lumping all AI bots together is a quick way to cause unintended drops in your search visibility.

• Standard Search Engines: Crawlers like Googlebot and Bingbot. You want these guys crawling your sitemaps and pages without any friction.
• AI Search and Assistants: Bots like PerplexityBot or OAI-SearchBot (OpenAI's search crawler). These fetch live data to answer user questions and link back to sources. If you block them, you're taking your site out of the running to be cited in generative search results.
• AI Training and Data Scrapers: Bots like GPTBot, CCBot, Anthropic-ai, and Google-Extended. These crawlers download your entire site just to build offline datasets and train large language models. They chew through massive amounts of bandwidth and send zero traffic back your way.

Building the robots.txt Policy

Your first line of defense is simply stating your rules out loud. The major corporate bots actually respect robots.txt, so you'll want to place this file right in the root directory of your web server (so it's accessible at https://yourdomain.com/robots.txt).

# Allow Standard Search Engines
User-agent: Googlebot
Allow: /
User-agent: Bingbot
Allow: /

# Block AI Training and Scraping Bots
User-agent: GPTBot
Disallow: /
User-agent: ChatGPT-User
Disallow: /
User-agent: Google-Extended
Disallow: /
User-agent: CCBot
Disallow: /
User-agent: Anthropic-ai
Disallow: /
User-agent: Omgilibot
Disallow: /
User-agent: Omgili
Disallow: /
User-agent: Cohere-ai
Disallow: /
User-agent: FacebookBot
Disallow: /

# Global fallback for unlisted bots
User-agent: *
Allow: /

Enforcing Rules at the WAF or CDN Level

Since robots.txt works entirely on the honor system, sketchy scrapers and customized headless browsers will completely ignore it. To actually save your server resources, you need to enforce rules at the edge so that unwanted traffic gets dropped before it ever reaches your origin server.

If you're using Cloudflare, you can set up Custom Rules to block known AI scrapers while leaning on their "Verified Bot" flag. That flag ensures legitimate search engines still get through safely, even if they happen to share IP subnets with noisy traffic.

Cloudflare WAF Expression Snippet

Head over to Security > WAF > Custom rules in your Cloudflare dashboard. Create a new rule, click "Edit expression", and paste in the following logic to catch AI scrapers without blindly trusting User-Agent strings alone:

(cf.bot_management.verified_bot eq false and (http.user_agent contains "GPTBot" or http.user_agent contains "Google-Extended" or http.user_agent contains "CCBot" or http.user_agent contains "Anthropic-ai" or http.user_agent contains "Cohere-ai"))

Set the action to Block. This rule looks for known AI User-Agents, while cf.bot_management.verified_bot eq false acts as a built-in safety net. Even if a legitimate, Cloudflare-verified search crawler somehow triggers a weird string match, it won't get blocked.

Verifying a Claimed Googlebot

Scrapers love to spoof their User-Agent strings, pretending to be Googlebot just to slip past your WAF rules and scrape locked-down content. Taking User-Agent: Googlebot at face value in your server logs leaves a pretty big hole in your defenses.

If an IP address is hammering your server and claiming to be Google, you can check it against Google's published googlebot.json IP ranges, or just run a quick reverse DNS lookup followed by a forward DNS lookup to be sure.

1. Reverse DNS Lookup: Run the host command on the suspicious IP address in your terminal.

host 66.249.66.1

Expected Output: 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com. Make sure the domain ends strictly with googlebot.com or google.com.

2. Forward DNS Lookup: Run the host command again, this time using the domain name you just got back, to confirm it points right back to the original IP.

host crawl-66-249-66-1.googlebot.com

Expected Output: crawl-66-249-66-1.googlebot.com has address 66.249.66.1

If both lookups match up cleanly, it's the real Googlebot. If the reverse DNS spits back a residential ISP, AWS, or DigitalOcean hostname, you're dealing with a fake scraper, go ahead and throw that IP block straight onto your WAF deny list.

How to Verify the Result

Once your robots.txt file and WAF rules are live, you'll want to keep an eye on your logs to make sure everything is working the way it should.

If you're running an Nginx origin server, you can tail your access logs and filter for the bots you just blocked to verify they aren't making it to your backend anymore:

tail -f /var/log/nginx/access.log | grep -E -i "GPTBot|Anthropic-ai|CCBot|Google-Extended"

If nothing shows up in your origin logs for those strings, your WAF is successfully dropping them at the edge. Next, double-check that normal Google indexing hasn't been interrupted by filtering for the real Googlebot:

tail -f /var/log/nginx/access.log | grep -i "Googlebot"

You should still see regular GET requests coming through with 200 OK status codes for Googlebot. Finally, pop into Google Search Console under Settings > Crawl stats. Give it about 48 hours for the data to catch up, then make sure your total crawl requests for Googlebot look steady and free of sudden spam URL spikes (such as those seen during a WordPress Japanese keyword hack), that's your confirmation that blocking AI scrapers hasn't messed with your search indexing.