Robots.txt for AI Bots: How to Control AI Search, Agents and Training Crawlers

The robots.txt file is one of the first places website owners use to communicate crawl preferences. As AI search engines, model-training crawlers, and user-triggered agents become more common, a single rule for all bots is no longer sufficient.

A well-designed robots.txt policy can allow AI search visibility while restricting model-training crawlers. However, robots.txt is advisory for compliant bots and must be combined with CDN, WAF, authentication, and rate-limiting controls when enforcement matters.

Where Is the Robots.txt File Located?

The file must be available at the root of the hostname:

https://example.com/robots.txt

A robots.txt file on www.example.com does not automatically control api.example.com, shop.example.com, or another subdomain.

Basic Robots.txt Structure

User-agent: ExampleBot
Disallow: /private/
Allow: /public/

The user-agent identifies the crawler, while Allow and Disallow define paths the crawler may or may not request.

Allow AI Search but Block Training

A site may choose to allow a crawler used for AI search while disallowing a separate crawler used for model training.

User-agent: OAI-SearchBot
Allow: /

User-agent: GPTBot
Disallow: /

This example communicates two different preferences: public content may appear in OpenAI search experiences, but it should not be crawled by GPTBot for foundation-model training.

User-Triggered AI Agents

User-triggered agents may fetch a page because a person requested current information. Their behavior may not match traditional background crawling. Some providers state that robots.txt treatment may differ for explicitly user-initiated requests.

Security teams should decide whether public pages should be accessible to verified user agents and whether login, checkout, account, or administrative paths require stronger controls.

Common Robots.txt Mistakes

• Blocking every bot with User-agent: *
• Editing robots.txt on the wrong hostname
• Returning 403 or 500 errors for robots.txt
• Blocking CSS or JavaScript needed for rendering
• Assuming robots.txt removes URLs from search results
• Using robots.txt to protect confidential data
• Forgetting that a CDN may cache the old file

Robots.txt Is Not a Security Boundary

Do not place confidential URLs in robots.txt as a protection method. The file is public and may reveal paths to anyone who reads it.

Use authentication, authorization, private networking, signed URLs, or firewall rules to protect sensitive information.

How DNS Changes Affect Robots.txt

When a domain moves to a new hosting provider or CDN, the robots.txt file may change unintentionally. The new server may return a default file, a redirect loop, a challenge page, or an access-denied response.

After changing DNS, verify the file from multiple networks and check the SSL certificate:

curl -I https://example.com/robots.txt

Use DNSCheckers.com Full DNS Lookup to confirm that crawlers reach the intended infrastructure.

How to Test AI Bot Rules

1. Open robots.txt in a normal browser.
2. Confirm the correct hostname and HTTPS response.
3. Review the exact user-agent spelling.
4. Check CDN and WAF logs for crawler requests.
5. Verify whether the crawler receives 200, 403, 429, or 5xx responses.
6. Confirm that allowed pages are not blocked by another rule.
7. Recheck after CDN cache expiration.

Recommended AI Robots.txt Strategy

• Allow crawlers that support the discovery channels you want
• Block training crawlers when training use is not permitted
• Restrict high-cost paths such as internal search pages
• Keep public documentation crawlable
• Use server-side controls for actual enforcement
• Review bot policies whenever providers change user agents

Frequently Asked Questions

Can a crawler ignore robots.txt?

Yes. Responsible crawlers generally comply, but malicious or poorly designed bots may ignore the file.

Does Disallow remove a page from search?

Not necessarily. A URL may still be discovered through links. Use noindex or authentication when removal is required.

How quickly do AI crawlers recognize a robots.txt change?

Timing varies by provider. CDN caching and the crawler's own refresh schedule can delay the effect.

Conclusion

robots.txt remains useful, but AI traffic requires more precise rules. Separate search, agent, advertising, and training crawlers, then combine robots.txt with verified bot controls and security policies.