OpenAI Crawler Guide: OAI-SearchBot, GPTBot, ChatGPT-User and OAI-AdsBot
OpenAI uses different web user agents for different purposes. Website owners should not treat OAI-SearchBot, GPTBot, ChatGPT-User, and OAI-AdsBot as interchangeable because each represents a separate use case.
What Is OAI-SearchBot?
OAI-SearchBot is used for search. Allowing it can help public pages appear in ChatGPT search results and cited answers.
User-agent: OAI-SearchBot
Allow: /Blocking OAI-SearchBot can prevent a site from being included as a normal search source, although navigational references may still be possible in some situations.
What Is GPTBot?
GPTBot is associated with crawling content that may be used to improve or train OpenAI's generative AI foundation models. A website can disallow GPTBot independently from OAI-SearchBot.
User-agent: GPTBot
Disallow: /What Is ChatGPT-User?
ChatGPT-User is used for certain user-triggered actions. A user may ask ChatGPT to open or retrieve information from a webpage. This traffic is not the same as automated search indexing.
Because the request is initiated by a person, its crawl and access behavior may differ from background crawlers. Sensitive pages should still be protected through authentication rather than robots.txt.
What Is OAI-AdsBot?
OAI-AdsBot is used to validate landing pages submitted for advertising and to evaluate their safety and relevance. Advertisers should ensure that approved landing pages are reachable by OAI-AdsBot.
User-agent: OAI-AdsBot
Allow: /landing-pages/Recommended Robots.txt Examples
Allow Search, Block Training
User-agent: OAI-SearchBot
Allow: /
User-agent: GPTBot
Disallow: /Allow Search and Ads
User-agent: OAI-SearchBot
Allow: /
User-agent: OAI-AdsBot
Allow: /
User-agent: GPTBot
Disallow: /Why OpenAI Crawlers May Receive 403 Errors
- Cloudflare or another CDN blocks automated traffic
- A WAF rule challenges unknown user agents
- robots.txt disallows the path
- The page requires cookies or JavaScript challenges
- Rate limiting returns HTTP 429
- Geo-blocking denies the crawler region
- The DNS record points to an old server
How to Verify an OpenAI Crawler
Do not trust the user-agent string alone. OpenAI publishes IP-range files for its crawler categories. Verification can combine:
- Exact user-agent matching
- Published IP ranges
- Verified-bot features from a CDN
- Firewall logs
- Request behavior and rate patterns
DNS Troubleshooting for OpenAI Crawler Access
- Check A and AAAA records.
- Verify that www and root hostnames use the intended service.
- Check CNAME targets behind the CDN.
- Confirm the SSL certificate.
- Verify that robots.txt returns HTTP 200.
- Review WAF events for the relevant user agent.
Use the DNSCheckers.com SSL Checker when the crawler reports certificate or hostname errors.
Frequently Asked Questions
Can I allow ChatGPT search but block training?
Yes. OAI-SearchBot and GPTBot are controlled independently.
Does allowing OAI-SearchBot guarantee citations?
No. It permits crawling for search but does not guarantee ranking, selection, or citation.
Should advertisers allow OAI-AdsBot?
Yes, when landing pages are submitted for ChatGPT advertising review.
Conclusion
OpenAI's crawler ecosystem separates search, training, user-triggered access, and advertising validation. Configure each user agent according to its actual purpose, verify legitimate traffic, and monitor DNS, CDN, WAF, and server logs.