Here is the number that prompted this piece. Alli AI analysed 24,411,048 proxy requests across 69 customer websites between 14 January and 9 March 2026, and reported that OpenAI’s ChatGPT-User made 133,361 requests against Googlebot’s 37,426. That is 3.6 times Googlebot’s volume, and more than Googlebot, Amazonbot and Bingbot combined. The analysis ran on Search Engine Journal as sponsored content written by Alli AI’s own chief executive, so read it as a vendor dataset rather than independent research. It is still a useful prompt, because many technical SEO audits size infrastructure thresholds around Googlebot’s volume, and a fetcher running at a multiple of that deserves a hard look at the rules sitting in front of your origin server.

Where the Policy File Stops and the Server Starts

Robots.txt is a request, not an enforcement mechanism. It tells a well-behaved crawler what it may fetch, and it says nothing to the web application firewall, the rate limiter in front of PHP-FPM, or the CDN’s bot-management ruleset, none of which read it. In many stacks those three layers were configured with Googlebot’s behaviour in mind. Googlebot adjusts its own pace in response to server health, which is part of why thresholds tuned around it have felt safe for so long. Google retired the crawl rate limiter tool in Search Console on 8 January 2024, citing improvements to its crawling logic and noting that Googlebot already slows itself down when a server returns errors or responds slowly. A newer class of AI fetchers, with its own user agents and request patterns, was not part of that tuning in many configurations.

OpenAI’s own crawler documentation splits its traffic across several user agents. GPTBot crawls content that may be used to train its foundation models. OAI-SearchBot surfaces websites in ChatGPT’s search features, and OpenAI recommends allowing it in robots.txt and also allowing requests from its published IP ranges. ChatGPT-User visits a page when a user’s question or a Custom GPT action calls for it, and OpenAI states that robots.txt rules may not apply to it because a person initiated the action. The same page says ChatGPT-User is not used to decide whether content may appear in Search, and that OAI-SearchBot is the one to manage in robots.txt for that purpose.

Read that carefully, because it changes the audit question. GPTBot, the crawler most people check in robots.txt, is the least relevant of the three to live retrieval. OAI-SearchBot, which decides whether you appear in ChatGPT search answers, is easy to overlook. And ChatGPT-User, the fetcher behind the Alli figures, may not consult robots.txt at all, which makes your firewall and CDN rules the main lever you hold over that traffic.

The wider picture is moving quickly and does not point in one direction. Botify and Nectiv analysed roughly seven billion OpenAI bot log events from enterprise sites between November 2024 and March 2026, and found that OpenAI’s automated crawling roughly tripled after GPT-5 launched in August 2025. OAI-SearchBot activity rose about 3.5x and GPTBot about 2.9x. ChatGPT-User moved the other way, falling 28% between December 2025 and March 2026 in that dataset, and even after the tripling, OpenAI’s crawlers came to about 4% of Googlebot’s volume. Two datasets paint two very different pictures, which is exactly the point. Alli’s sample is small, WordPress-heavy and drawn from customers who opted into a crawler enablement platform, while Botify’s skews towards large enterprise sites. Neither is a portrait of your site.

My Personal Anecdote

I run LadyinTechverse on managed WordPress hosting at Verpex, after starting out self-hosting on DigitalOcean, and robots.txt was always the last line of my crawler checks. Nothing in it ever looked wrong, which is exactly the problem. After an audit, I realised a clean file says nothing about what reaches the origin. I honestly did not know whether my own site was turning ChatGPT away at the door. My robots.txt says it is welcome, but that file is only a sign on the gate. Until I checked the records of who actually got through, all I have is a hunch.

Why the Audit Checks One File and Stops

Many technical SEO audits, including ones I have run on WordPress, first self-hosted on DigitalOcean and now on managed hosting at Verpex, treat robots.txt as the finish line for crawler access control. Confirm nothing important is disallowed, tick the box, move to the next line item. That habit comes from a decade in which Googlebot was the only crawler that mattered at meaningful volume, and a clean policy file was a reasonable proxy for access.

AI fetchers break that proxy, because every user agent has its own rules and the infrastructure in front of your origin classifies bots on its own terms. Take a rate limiter set at 60 requests per minute per IP range for bot traffic. That figure is purely illustrative, and whether it would ever catch an OpenAI fetcher depends on how those requests arrive, in what bursts, and from which published ranges. Public data cannot answer that for your site. Averaged across Alli’s sample, ChatGPT-User made roughly 35 requests per site per day, which is my own arithmetic from the published totals and which hides whatever spread exists underneath. In the same dataset ChatGPT-User received a successful response 99.99% of the time, but Alli serves pre-rendered pages from the CDN edge, a very different set-up from a self-hosted WordPress stack. The sample shows what is possible, not what your origin and firewall would do.

When a block does happen, it is usually recorded in the firewall or CDN’s own logs, and it may never appear in the origin server log at all. That is why the log you check matters as much as the rule you read. I covered the allow-or-block decision itself, the GEO policy layer, in an earlier piece on AI crawler access control. This piece is about the layer underneath that decision, the one that could quietly reverse it.

Cloudflare’s Bot Preference Sync, announced on 21 August 2026 and covered by Slobodan Manic in a column for Search Engine Journal on 18 September, generates a robots.txt file from three dashboard settings for Search, Agent and Training bots. Cloudflare’s stated rationale, as Manic reports it, is that when a site’s declared preferences and its enforced rules disagree, some crawlers treat that gap as grounds to disregard the preferences. I read that as a tacit acknowledgement from the infrastructure side that policy and enforcement drift apart. Manic’s own objection is that three categories cannot express a per-crawler policy, and he notes that at the time of writing the feature had not yet appeared in Cloudflare’s bots changelog, so treat it as announced rather than universally live.

The Four-Layer Audit That Confirms Access

A technical SEO audit that wants to know whether OpenAI’s fetchers can reach a site, rather than whether the policy file welcomes them, needs to inspect four layers in order. First, robots.txt itself. Confirm GPTBot and OAI-SearchBot are treated the way you intend, remember that ChatGPT-User may not follow the file at all, and check the directive syntax against OpenAI’s documentation, since a malformed user-agent match is its own silent block. Second, the WAF ruleset. Pull the rate-limit and bot-management rules by name rather than by assumption, and identify which request-per-minute or request-per-hour ceiling applies to the IP ranges and user-agent strings OpenAI publishes for each of its bots. Third, the CDN layer, which on many self-hosted WordPress stacks sits in front of the WAF and makes its own bot-classification decision before traffic ever reaches the origin. Fourth, the logs. Start with the WAF and CDN logs, then check the origin server logs, and look for a run of 403 or 429 responses against OpenAI’s published ranges, because that is the tell that policy said yes while infrastructure said no.

This fourth layer matters because it cannot be faked by good intentions. A marketing leader can confirm robots.txt is correct and still have little ChatGPT traffic reaching the origin, because the WAF may have returned a 403 before the origin log line was ever written. Auditing only the policy file can produce false confidence. The audit passes, the citation eligibility it was meant to protect does not materialise, and nobody connects the two because the failure happened one layer below where anyone looked.

What This Costs a B2B Marketing Leader Who Never Audits It

GEO and AEO visibility depend on AI systems being able to retrieve current content, not on a policy document saying they are welcome to. A B2B site that has invested in structured data, clean chunking and citation-worthy answer formatting, the work I have written about in the context of retrieval failure modes behind wrong AI agent citations, gets non of that benefit if the fetcher carrying the content back to the model is throttled or blocked at the edge. I cannot show from public sources that this is happening on any particular site, and that is exactly why the audit matters. From the infrastructure side, blocking an aggressive, high-volume bot is simply a rate limiter doing its job, so the content team never hears about it and the infrastructure team never thinks to flag it.

This also compounds the measurement blind spot covered in why AI agent traffic hides inside Chrome analytics. A marketing leader who cannot see AI agent traffic in GA4, and whose infrastructure team may be rate-limiting that same traffic is working from two independently incomplete pictures, each one making the other look less urgent than it is.

Final Thoughts: The Bottom Line

Robots.txt is policy. WAF rules, rate limiters and CDN bot management are enforcement. For ChatGPT-User, OpenAI’s own documentation says the policy file may not even apply, which makes enforcement the layer that matters most. The two sets of controls are configured by different teams, reviewed on different schedules and audited by different checklists, which is how a site can say yes at the policy layer and no at the infrastructure layer without anyone noticing the contradiction.

The fix is not a bigger robots.txt file. It is a technical SEO audit that treats crawler access control as a four-layer question, with logs as the layer that tells the truth about what the other three did. The datasets cited here cannot tell you whether your stack is blocking OpenAI’s fetchers. Only your own logs can. If AI citation eligibility matters to your marketing outcomes in 2026, this is an audit line item worth adding. Read how I approach that work.

Frequently Asked Questions (FAQs)

Does allowing GPTBot in robots.txt guarantee ChatGPT can reach my site?

No. GPTBot is OpenAI’s training crawler. Surfacing sites in ChatGPT search is handled by OAI-SearchBot, and user-initiated page visits are handled by ChatGPT-User. Each has its own rules, and a firewall, rate limiter or CDN can block any of them regardless of what robots.txt says.

Does robots.txt control ChatGPT-User?

OpenAI’s documentation says that because ChatGPT-User actions are initiated by a user, robots.txt rules may not apply. It also says ChatGPT-User does not decide whether content appears in Search, and that OAI-SearchBot is the agent to manage in robots.txt for that. For ChatGPT-User, server-level rules are the control you have.

Why might a WAF or rate limiter affect OpenAI crawlers when robots.txt allows them?

WAF, rate-limiter and CDN rules do not read robots.txt. Alli AI’s sponsored analysis of 69 sites found ChatGPT-User making 3.6 times Googlebot’s requests, so thresholds tuned around Googlebot may behave differently with that traffic. No public dataset shows how often blocking actually happens, which is why you should check your own rules and logs.

How do I check whether my site is blocking OpenAI’s crawlers?

Pull your WAF and CDN bot-management rules by name, compare their rate-limit ceilings with the IP ranges OpenAI publishes for each bot, then check WAF, CDN and origin logs for 403 or 429 responses against those ranges. A cluster of them means enforcement is overriding policy.

Is this the same issue as deciding whether to allow or block AI crawlers?

No. The allow-or-block decision is a GEO policy choice. This is the separate infrastructure layer, made up of rate limiters, WAF rules and CDN thresholds, that can quietly reverse an allow decision after it has been made.

Why does this matter for AI citation eligibility?

GEO and AEO visibility depend on AI systems being able to retrieve your content. OAI-SearchBot is the agent OpenAI says surfaces sites in ChatGPT search, so it is the first one to verify. Content investment in structured data and clean formatting delivers no citation benefit if the fetcher is blocked before it reads the page.