Cloudflare's Pay-Per-Crawl model introduces HTTP 402 (Payment Required) as a mechanism for publishers to charge AI systems for accessing their content. For most brand websites, this is a trap disguised as an opportunity. If you gate your content from AI crawlers, AI platforms cannot cite you, and you disappear from the recommendation layer that is now shaping buyer decisions. The brands that should consider Pay-Per-Crawl are those whose content is their core product: paywall publications, proprietary databases, premium research platforms. For everyone else, the correct GEO strategy is to actively ensure your content is crawlable, visible, and structured for citation. This guide explains the decision framework and the specific robots.txt settings that maximize your AI citation potential.
The question of whether AI platforms should be allowed to scrape web content has been building for years. Publishers have watched their content get used to train AI models and power AI answers without compensation or attribution. Cloudflare's Pay-Per-Crawl model is the first infrastructure-level response: a technical mechanism that lets any publisher charge AI crawlers for access, or block them entirely, without needing to negotiate individual licensing deals.
From a publisher rights perspective, this is a meaningful development. From a brand GEO perspective, it raises an immediate strategic question: if we can monetize AI access to our content, should we? And what happens to our AI visibility if we choose to block or charge crawlers who do not pay?
The answer matters more than most marketing teams realize. In my experience, a significant number of brands have accidentally blocked major AI crawlers through overly restrictive robots.txt files, security settings, or Cloudflare rules configured for other purposes. They have no idea. Their content is invisible to the AI platforms their buyers use daily, and nothing in their analytics tells them why.
How Cloudflare Pay-Per-Crawl Works
The HTTP 402 status code has been defined in the HTTP specification since 1996 as "Payment Required" but was never widely implemented. Cloudflare's Pay-Per-Crawl is the first major infrastructure deployment of this code at scale. When a publisher activates the feature, any AI crawler that hits their site receives a 402 response with payment instructions. AI companies that have pre-negotiated Cloudflare micropayment credits can automatically pay and proceed. Others are blocked unless they complete the payment flow.
Publishers set their own price per crawl request. The range in early deployments runs from fractions of a cent for standard content to several cents for premium or specialized content. AI companies decide whether the content is worth the cost. For large training data crawls, even tiny per-request costs add up quickly across millions of pages, which means expensive content is likely to be skipped in favor of freely available alternatives.
For real-time AI search platforms like Perplexity, the economics are different: they need fresh content continuously, which means they are more likely to pay for high-value sources than to skip them. But they are also more likely to develop their own partnerships with publishers rather than paying per-crawl at scale.
The three options every publisher now has
- Open access (current default for most brands): Your content is freely crawlable by all AI systems. You receive no payment but maximum AI citation potential. Correct for almost all brand marketing websites.
- Pay-per-crawl via Cloudflare: You charge AI crawlers for access. You potentially earn micropayment revenue. You risk reduced AI citation frequency if crawlers skip you. Correct for content-as-product publishers.
- Full block via robots.txt: You refuse all AI crawler access. You earn nothing and receive zero AI citations. Only defensible if you are protecting proprietary data that competitors could use against you, and even then, the citation cost is high.
The Robots.txt Audit Every Brand Needs to Run
Before engaging with the Pay-Per-Crawl decision at all, most brands have a more urgent problem: they have never checked whether their existing robots.txt file accidentally blocks AI crawlers. This is more common than you might expect. Security-focused developers often add blanket bot-blocking rules. Legacy robots.txt files from five years ago predate the existence of the major AI crawlers. Cloudflare security rules sometimes block crawlers that look like automated traffic.
The crawlers you need to actively allow for maximum AI citation coverage are listed below. Check each against your robots.txt and your server-side access logs.
| AI Platform | Crawler Name(s) | Citations Generated |
|---|---|---|
| ChatGPT (web search) | GPTBot, ChatGPT-User, OAI-SearchBot | ChatGPT search answers |
| Gemini / Google AI Mode | Googlebot, Google-Extended | Gemini, Google AI Mode answers |
| Perplexity | PerplexityBot | All Perplexity citations |
| Claude (web access) | ClaudeBot, anthropic-ai | Claude web-search citations |
| Grok | xAI-Bot | Grok web search citations |
If any of these are blocked in your robots.txt, you are invisible to that platform's citation engine. Fix it today. The SEO cost of allowing these crawlers is zero. The GEO cost of blocking them is total invisibility on that platform.
Jeevan AI audits your crawler accessibility alongside your citation performance across every major AI platform.
The Decision Framework: Should Your Brand Use Pay-Per-Crawl?
Use this framework to decide whether Pay-Per-Crawl makes sense for any specific content on your site.
- Is the content your core product? If your content is what customers pay you for (a news publication, a research database, a premium insights platform), charging AI crawlers for access is defensible and potentially valuable.
- Would being cited by AI drive more value than the micropayment? For most brand marketing content, a single citation that influences a buyer's research process is worth far more than the few cents a Pay-Per-Crawl transaction would generate. Choose citations.
- Do you have proprietary data competitors could exploit? If your site contains detailed operational data that competitors could use to train models against you, restricted access may make sense for those specific pages. For general marketing and educational content, it does not.
- Are you already well-cited and looking to monetize? If your brand already has strong AI SoV and you are being cited extensively by AI platforms, exploring Pay-Per-Crawl as an incremental revenue stream is reasonable. Do not experiment with it before you have established citation presence.
Frequently Asked Questions
What is Cloudflare's Pay-Per-Crawl model?
Cloudflare's Pay-Per-Crawl uses the HTTP 402 (Payment Required) status code to allow website owners to charge AI crawlers for accessing their content. When an AI crawler encounters a 402 response, it can either complete a micropayment to access the content or move on. Publishers set the price per crawl request, and AI companies that want to use that content must pay. It is an alternative to blocking AI crawlers entirely or allowing free access.
Should I block AI crawlers from my website?
For most brands, blocking AI crawlers is a significant mistake from a GEO perspective. If AI platforms cannot crawl your content, they cannot cite it. Brands that block Perplexity's crawler will never appear in Perplexity citations regardless of how good their content is. The only situation where blocking makes clear business sense is when your content is your core product and you want to monetize AI access rather than give it away. For most brand websites, the value of AI citations far exceeds the hypothetical value of charging for crawl access.
Which AI crawlers should I allow in my robots.txt?
The key AI crawlers to allow are: PerplexityBot (for Perplexity citations), Googlebot and Google-Extended (for Gemini and Google AI Mode), GPTBot and ChatGPT-User (for ChatGPT web search), ClaudeBot (for Claude's web access), and xAI-Bot (for Grok). Blocking any of these eliminates your citation potential on that specific platform. Verify your current robots.txt allows these crawlers by checking your file against the official crawler names each company publishes.
The Pay-Per-Crawl economy is coming regardless of whether individual brands choose to participate. The right response for most brand marketing teams is not to engage with the monetization question at all, but to treat it as a wake-up call for a basic crawl accessibility audit. Most brands have never checked whether major AI crawlers can actually access their content. Run the audit this week.
Once you have confirmed your content is accessible to the crawlers that matter, the GEO investment is straightforward: structure your content for citation, publish consistently, and measure your AI SoV monthly. That program builds durable AI visibility that no Pay-Per-Crawl mechanism can take away.